Cambridge Commentary on EU General-Purpose AI Law

2. Substance

2.1. Article 55(1): Additional obligations for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

11Article 55(1) introduces a set of substantive obligations for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . These obligations apply in addition to those already imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models under Article 53¹³ and, where applicable, Article 54 AI Act.¹⁴ The inclusion of these supplementary obligations is justified on the basis that GPAI models with systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may give rise to ‘potential significantly negative effects’ that exceed those associated with general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. more broadly.¹⁵ The severity of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. – defined as a risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. capable of propagating at scale across the AI value chain and producing significant impact on the Union market, including through society-wide harm – thus warrants the inclusion of commensurate obligations, in line with the AI Act’s underlying risk-based logic.¹⁶ In addition to the severity of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , Article 55(1) also responds to the inherent unpredictability of the most advanced GPAI models. Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is understood to increase with model capabilities,¹⁷ which in turn may only emerge and become apparent after market placement, in ways that cannot be predicted or anticipated at the pre-deployment stage.¹⁸ This creates what may be characterised as an epistemic problem as much as a severity problem: the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. are not only uniquely severe but are inherently difficult to anticipate and mitigate, even by those who develop the models.¹⁹ GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. therefore present a fundamentally different type of regulatory challenge,²⁰ which in turn justifies the imposition of the additional obligations under Article 55(1).

12The nature of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. also informs the interpretation of the obligations outlined in Article 55(1),²¹ which are to be read in light of the objective to assess and mitigate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ²² with a degree of scrutiny and level of detail proportionate to the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. involved.²³ It should be underscored that the purpose of the obligations under Article 55(1) is not to prevent systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. from materialising altogether, but rather to achieve ‘a comprehensive prevention in the sense of minimising the probability of occurrence as much as possible and preparing as effectively as possible should foreseen or unexpected systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. materialise.’²⁴ The obligations apply along the model’s entire lifecycle and regardless of whether the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is provided as a standalone model or embedded in an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. or a product.²⁵ This means providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must take appropriate measures along the entire model’s lifecycle and cooperate with relevant actors along the AI value chain;²⁶ the fact that the downstream system complies with the corresponding regulatory requirements does not release the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. from fulfilling its obligations under Article 55.²⁷ 

13The provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. obligations are outlined across the four subparagraphs of Article 55(1). Under Article 55(1)(a), providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must conduct model evaluations using ‘standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.²⁸ Article 55(1)(b) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ‘assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level, including their sources, that may stem from the development, the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , or the use’ of the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²⁹ Pursuant to Article 55(1)(c), providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must ‘keep track of, document, and report […] relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. and possible corrective measures to address them’.³⁰ Finally, Article 55(1)(d) obliges providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ‘to ensure an adequate level of cybersecurity protection’ for both the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and its physical infrastructure.³¹ 

14Each of these obligations will be analysed in turn, following the order of the provision, although the relationship between the obligations themselves is not necessarily sequential. The obligations in Article 55(1) are continuous,³² meaning that the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process must be revisited along the model’s lifecycle as circumstances change.³³ The obligations are also iterative, meaning that the steps of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process are cyclical and repeat themselves, feeding back into one another, at least until all risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. have been reduced to an acceptable level.³⁴ Although Article 55 does not explicitly describe the obligations as such – a notable omission given that the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process for high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 9 is expressly characterised as a ‘continuous iterative process’³⁵ – iteration is nonetheless a central feature of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management.³⁶ 

15This understanding of the obligations as continuous and iterative therefore means that the four subparagraphs should be read as interacting with and reinforcing one another, rather than as discrete and self-contained requirements.³⁷ This is particularly evident in the relationship between Article 55(1)(a) and 55(1)(b). Article 55(1)(a) focuses on the use of state-of-the-art model evaluations with a view to identifying systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . However, risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification itself constitutes the first stage of the established risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management pipeline reflected in international standards and seemingly invoked by the EU legislature elsewhere in the AI Act, most notably in the context of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .³⁸ Accordingly, the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification obligation under Article 55(1)(a) should be understood as forming part of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation obligations imposed under Article 55(1)(b).

16Similarly, the obligations under Article 55(1)(c) concerning the tracking, documenting, and reporting of relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. may contribute to identifying previously unrecognised failure modes or expose the inadequacy of existing mitigation measures.³⁹ Article 55(1)(d), in turn, introduces cybersecurity obligations aimed at protecting the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and its physical infrastructure against malicious interference or compromise.⁴⁰ Although such measures should be distinguished from safety-oriented mitigation measures directed at preventing harmful outcomes arising from the model’s capabilities or behaviour,⁴¹ they nonetheless remain part of the broader systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. management framework established under Article 55. Accordingly, a systematic and purposive reading supports interpreting the obligations under Article 55(1)(a) through (d) as constituting interconnected elements of a continuous systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. management process, beginning with risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification and extending to risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. analysis,⁴² evaluation,⁴³ and (safety and security) mitigation.⁴⁴ Such a framing is also consistent with how the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. management process is envisaged in the Safety and Security Chapter of the GPAI Code of Practice.⁴⁵ 

17In this vein, it is necessary to address the interpretative value of the Safety and Security Chapter of the GPAI Code of Practice. The Safety and Security Chapter was drafted specifically in relation to the obligations imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. under Article 55(1) AI Act.⁴⁶ Although the commitments and accompanying measures contained therein do not expressly distinguish between the individual subparagraphs of Article 55(1), the present chapter will, where appropriate, broadly map them onto the corresponding obligations contained in Article 55(1)(a) through (d). At the same time, it must be emphasised that the Code of Practice remains a voluntary and therefore non-binding instrument.⁴⁷ The commitments and measures contained therein do not themselves impose legal obligations on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. and therefore cannot determine the meaning of Article 55(1) as a matter of law.⁴⁸ 

18Nevertheless, the interpretative significance of the GPAI Code of Practice should not be understated. The fact that the European Commission has deemed the Safety and Security Chapter as an acceptable means through which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may demonstrate compliance with Article 55(1) lends the commitments and measures contained therein particular practical and interpretative relevance.⁴⁹ In that respect, the Code may be regarded as indicative of how the Commission understands the operationalisation of the obligations imposed under Article 55(1) and thus as carrying interpretative weight for a purposive reading of the provision.⁵⁰ The Code of Practice likewise plays a crucial role in the Commission’s assessment of ‘alternative adequate means’ that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that do not adhere to the Code might implement – underlining the Code’s de facto key role in interpreting the obligations in Article 55(1).⁵¹

19This chapter identifies ambiguities arising from the text of Article 55(1), as well as uncertainties concerning its practical operationalisation, and analyses the strongest interpretative approaches to resolving them through textual, systematic, and teleological interpretation of the AI Act. Within that interpretative framework, the Safety and Security Chapter of the GPAI Code of Practice serves as an important interpretative reference point and, in several instances, provides particularly compelling guidance for resolving ambiguities concerning the operationalisation of the obligations imposed under Article 55(1). At the same time, consistent with this commentary’s broader commitment to interpretive optionality, alternative interpretations are identified and discussed where they remain legally plausible.

2.1.1. Article 55(1)(a): Conducting state-of-the-art model evaluations, including adversarial testing

20Article 55(1)(a) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ‘perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model with a view to identifying and mitigating systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’. Recital 114 further supplies that ‘the necessary model evaluations’ are to be conducted ‘in particular prior to its first placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. ’, including through adversarial testing and, where appropriate, ‘through internal or independent external testing’.⁵²

21This section proceeds on the basis that Article 55(1)(a) primarily concerns the identification of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. through state-of-the-art model evaluations, while recognising its functional relationship with the broader systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation obligations imposed under Article 55(1)(b).⁵³ It should also be reiterated that other practices and techniques aside from model evaluations can be used for identifying systemic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .⁵⁴ Likewise, model evaluations may be used at multiple stages of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process, including during systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis and the development of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation measures.⁵⁵

22The analysis of Article 55(1)(a) proceeds through several closely related questions concerning the scope and operationalisation of the obligation. The first question concerns the obligation to conduct state-of-the-art model evaluations and in particular how the notion of state of the art should be understood as a benchmark for the appropriateness of the evaluations conducted in the context of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation. The analysis then turns to the concept of model evaluation itself, including the meaning of adversarial testing as a type of model evaluation expressly listed both in Article 55(1)(a) and the supporting Recital 114. A related question concerns whether Article 55(1)(a) contemplates only internal model evaluations or also requires independent external evaluations.⁵⁶ Finally, this section considers whether the obligation to conduct and document adversarial testing – as expressly required under Article 55(1)(a) – extends beyond the evaluations themselves to encompass broader documentation obligations relating to the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation process.

2.1.1.1. The ‘state of the art’ condition

23Article 55(1)(a) requires that model evaluations conducted by providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. reflect the state of the art. In this context, the state of the art functions as a dynamic reference that avoids fixing detailed technical requirements in the text of the provision and instead tethers the threshold of compliance to evolving scientific and technical knowledge.⁵⁷ The legally binding obligation therefore remains compliance with the statutory standard established under Article 55(1)(a),⁵⁸ while the concrete measures, methodologies, and evaluation techniques capable of satisfying that standard may evolve over time.⁵⁹ Outside of Article 55, the AI Act requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. to ensure compliance with applicable requirements by ‘taking into account their intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. as well as the generally acknowledged state of the art on AI and AI-related technologies’.⁶⁰ Any codes of practice and harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. developed as means for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to demonstrate conformity with the requirements of the AI Act are likewise expected to reflect the state of the art.⁶¹

24While the concrete risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures capable of satisfying the state-of-the-art condition may evolve over time, the reference itself cannot remain entirely indeterminate.⁶² A helpful starting point to contextualise the subsequent analysis is the three-stage theory developed by the German Federal Constitutional Court in the 1978 Kalkar I decision.⁶³ Although not binding on the EU institutions or on the interpretation of EU law,⁶⁴ the decision has served as an influential conceptual reference point for distinguishing between different levels of technological development reflected in legal standards.⁶⁵ The Court situated the state of the art between, on the one hand, the generally accepted rules of technology, reflecting the ‘prevailing opinion among technical practitioners’,⁶⁶ and, on the other hand, the state of scientific knowledge and research, encompassing the latest scientific and technical developments irrespective of their practical feasibility.⁶⁷ The generally accepted rules of technology have also been described as broadly corresponding to best practices.⁶⁸ Best practices are those measures that have proven themselves in practice and are hardly subject to methodological modernisation.⁶⁹ Best practices often serve as ‘the minimum basis for state of the art,’⁷⁰ or, put differently, that the ‘state of the art at least corresponds to best available [practices]’.⁷¹ 

25Within this framework, the state of the art occupies an intermediate position between generally accepted rules of technology and the state of scientific knowledge and research: it exceeds what is merely generally accepted or routinely implemented in practice but does not extend to the furthest frontier of scientific research and development.⁷² Nor are the boundaries between these three categories of technical standards impermeable. Measures that initially qualify as representing the state of scientific knowledge and research will, upon market introduction and subsequently being proven in practice, move into the category of state of the art.⁷³ With increasing standardisation, distribution, and market recognition, such measures may become widely deployed and routinely reflected in technical standards.⁷⁴ As a result, their level of innovation may decline as they come to be recognised as generally accepted rules of technology.⁷⁵ Distinguishing when a measure ceases to be state of the art and instead becomes a generally accepted rule of technology also remains challenging since general recognition and practical validation alone are not decisive.⁷⁶ Best practices can remain widely used even where their effectiveness has declined, whereas the state of the art excludes measures that no longer provide adequate safety assurance despite continued availability on the market or standardisation.⁷⁷

26Situating the state of the art within this broader spectrum of dynamic references is not merely conceptual. The position attributed to the state-of-the-art condition under Article 55(1)(a) directly influences the level of diligence expected of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. when conducting systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. evaluations and implementing corresponding mitigation measures. Two principal interpretative pathways are explored in this chapter.

27First, the state-of-the-art condition in Article 55(1)(a) could be understood in line with the definition provided in the Safety and Security Chapter of the GPAI Code of Practice – that is, ‘the forefront of relevant research, governance, and technology that goes beyond best practice’.⁷⁸ That definition was developed specifically in the context of Article 55 and has been endorsed by the European Commission as an adequate means for demonstrating compliance with the provision.⁷⁹ Alternatively, the state-of-the-art condition in Article 55(1)(a) could be interpreted in line with the systematic reading of the AI Act by reference to the generally acknowledged state of the art applied to high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 8(1).⁸⁰ Depending on which interpretation is adopted, and therefore where the state-of-the-art condition is situated along the spectrum between established best practices and the frontier of technological development,⁸¹ the provision will shape both the level of rigour, breadth, and depth expected of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. when conducting model evaluations and the degree of technical effort and innovation they are expected to invest in advancing AI safety and security practices more broadly.⁸²

2.1.1.1.1. State of the art as corresponding to the frontier of scientific knowledge and research

28As indicated, one approach to interpreting the state-of-the-art condition in Article 55(1)(a) is to locate it closer to the domain of scientific research and technological development, in a reading that would prioritise active innovation over measures that are merely generally recognised and proven in practice. Indeed, the definition of state of the art developed in the GPAI Code of Practice points toward this reading. The Safety and Security Chapter defines state of the art as ‘the forefront of relevant research, governance, and technology that goes beyond best practice.’⁸³ Best practices, in turn, are defined as the ‘processes, measures, methodologies, methods, and techniques […] accepted amongst providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as [those] that best assess and mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at any given point in time.’⁸⁴ Measures that qualify as state of the art would be those that ‘demonstrate equal or superior safety or security outcomes through alternative means that achieve greater efficiency [compared to approaches accepted as best practice].’⁸⁵ The EU regulator may therefore assess whether measures reflect the state-of-the-art condition not by reference to what is ‘generally recognised or established in practice, but what is technically necessary, appropriate and possible, even if commercial practice is not yet in line with it’.⁸⁶ 

29A purposive reading of Article 55(1)(a) likewise supports interpreting the state-of-the-art condition as referring to a more dynamic standard rather than merely one calibrated to industry practice. As the German Federal Constitutional Court recognised in Kalkar I, where legislation seeks to regulate technologically complex and rapidly evolving risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. :

it would not only be inappropriate, but actually contrary to the purpose of the regulation, if the legislature were to establish safety requirements through normative provisions oriented towards the status quo of technological development. […] A legislator who strives in this way to keep pace with technological developments, specifically in the interest of intensifying safety, can hardly be faulted.⁸⁷

30There are policy considerations that support this interpretation. Setting the state-of-the-art condition above best practices, and thus pursuing what was described above as state of the art at the forefront of technical advancement, creates conditions under which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are not only required to invest in safety but also rewarded for doing so. This incentive structure mirrors insights developed in the context of the state-of-the-art defence as found in US liability law, where it has been argued that ‘average safety increases when a state of the art defence is based on the technological advancement test.’⁸⁸ Under a regime in which the state of the art is equated with industry-wide best practices, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. would have little incentive to invest in costly safety improvements, as placing a safer model on the market does not reduce exposure to regulatory scrutiny or enforcement.⁸⁹ However, where the state of the art is linked to technical advancement, a subset of safety leaders may invest in demonstrably higher levels of protection, giving rise to a form of race-to-the-top ‘safety contest’ that rewards performance relative to peers.⁹⁰ While this theory was originally developed in the context of liability law, it can also be discerned through a teleological reading of the Code of Practice and its recitals, which emphasise the objective of fostering a culture of innovation in AI safety and security.⁹¹ 

31Compliance with the state-of-the-art condition under this interpretation is therefore not a one-off exercise but a continuous process.⁹² It requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to remain attentive to developments in the broader field of model evaluation and update accordingly.⁹³ Such developments could constitute reasonable grounds to question whether earlier assessments of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. remain valid.⁹⁴ Where such grounds arise, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who are signatories to the Code of Practice are therefore required to revisit their systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation process and update their Safety and Security Model Report accordingly within a reasonable amount of time.⁹⁵ Notably, the definition adopted in the Code of Practice, and the interpretation it endorses, are likely to have implications not only for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who are signatories to the Code of Practice but for all providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. subject to the obligations under Article 55.⁹⁶ It is, however, not implausible that a claim could be brought on the basis of a systemic interpretation of the AI Act, arguing that the state-of-the-art condition in Article 55(1)(a) should instead be read in line with the way state of the art has been defined in the context of Chapter III on high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . After all, appropriate legislative drafting demands substantive consistency across the legislation, meaning that defined terms must be used uniformly and that their content should not diverge from the definitions provided;⁹⁷ accordingly, the definition of state of the art should be consistent across the AI Act.

2.1.1.1.2. Systematic alignment of state of the art with the ‘generally acknowledged state of the art’

32An alternative approach to interpreting the state-of-the-art condition in Article 55(1)(a) is to understand it, through a systematic reading of the AI Act, as corresponding more closely to what the Regulation refers to as the ‘generally acknowledged state of the art’ in Article 8(1). The qualifier generally acknowledged was introduced during the legislative process to replace an earlier reference in the AI Act draft to the current state of the art.⁹⁸ The European Parliament had previously proposed to define the state of the art as ‘the level of development of technical capabilities at a given point in time with regard to products, processes and services, based on the relevant consolidated knowledge of science, technology and experience’.⁹⁹ While that definition retained a clear temporal and therefore dynamic element, tethering the standard to developments ‘at a given point in time’, the introduction of the qualifier ‘generally acknowledged’ arguably shifts the emphasis away from the frontier of technological development and towards methods and techniques that have achieved broader professional recognition and validation in practice.

33This understanding has also been reflected in commentaries on Article 8 AI Act,¹⁰⁰ where the ‘generally acknowledged state of the art’ has been distinguished from the latest ‘state of the art’ as developed by the most innovative industry leaders and instead associated with ‘the generally accepted level of technical risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. minimization among providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the same type of AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’.¹⁰¹ In effect, this interpretation brings the ‘generally acknowledged state of the art’ closer to what the Kalkar I framework described as the generally accepted rules of technology. In fact, the coherence of the standard of ‘generally acknowledged state of the art’ has itself been questioned on the basis that ‘there is no such thing as a standard of “generally accepted state of the art”’.¹⁰² Rather, the relevant standard may more plausibly refer either to the ‘state of the art’ or to the ‘generally accepted rules of technology’, as understood in the Kalkar I decision, but not to a combination of the two. Combining both standards risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. collapsing the distinction between measures that are merely established and validated in practice and those that reflect the current level of technological advancement.¹⁰³

34The AI Act reiterates that harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and GPAI models may rely upon to demonstrate compliance with the obligations must reflect the state of the art.¹⁰⁴ In its Draft Standardization Request in Support of Safe and Trustworthy Artificial Intelligence for high-risk systems, the European Commission subsequently defines ‘state of the art’ as meaning ‘a developed stage of technical capability at a given time […] based on the relevant consolidated findings of science, technology and experience and which is accepted as good practice in technology.’¹⁰⁵ In that context, the Commission goes further to say that ‘the state of the art does not necessarily imply the latest scientific research still in an experimental stage or with insufficient technological maturity.’¹⁰⁶ Compared to the definition of state of the art adopted in the Code of Practice, the definition of state of the art as interpreted under Chapter III thus corresponds more closely to what the Code of Practice defines as best practices, which describes measures that ‘best assess and mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at any given point in time.’¹⁰⁷ Indeed, where providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. do not rely on harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. that reflect the state of the art,¹⁰⁸ it is advisable for them to take account of ‘already known industry standards relating to AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. with similar purpose and algorithmic functionality, and document this accordingly.’¹⁰⁹ 

35If the state-of-the-art condition in Article 55(1)(a) were to be interpreted in line with the ‘generally acknowledged state of the art’ referred to in Chapter III, the obligation under Article 55(1)(a) could be described as more provider-friendly¹¹⁰ because it would be sufficient for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to follow the lead of comparable GPAI model providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. rather than respond to the measures adopted by the most innovative industry leaders.¹¹¹ This lateral comparison is particularly useful insofar as it reinforces the product-safety-oriented understanding of the state of the art underpinning the regulation of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . At the same time, however, there are important reasons to question whether equivalent interpretations should automatically be transposed to the GPAI model rules. For one, the extent to which the AI Act can be understood as product-safety legislation when applied to general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. remains contested,¹¹² particularly given that existing AI standards do not directly address GPAI models, thereby leaving open what constitutes state-of-the-art safety and security mitigations in this context.¹¹³ 

2.1.1.2. Model evaluations that reflect the state of the art

36Article 55(1) introduces model evaluations as a mechanism through which systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are to be assessed and mitigated. In the absence of an operative definition of model evaluation in the AI Act, reference may be made to the GPAI Code of Practice’s functional understanding of model evaluations.¹¹⁴ The latter defines model evaluations as a ‘ systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment technique that can be used at all stages of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment’, which in turn includes all methods of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification, analysis and acceptance determination.¹¹⁵ This definition is ostensibly broad, allowing for different methods and techniques provided that these are ‘appropriate for the model and the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.¹¹⁶ Adversarial testing is identified as one example of state-of-the-art model evaluations providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must conduct.¹¹⁷ Other types of model evaluation techniques acknowledged by the European Commission as suitable for the purposes of demonstrating compliance with Article 55(1)(a) include ‘Q&A sets, task-based evaluations, benchmarks, red-teaming and other methods of adversarial testing, human uplift studies, model organisms, simulations, and/or proxy evaluations for classified materials.’¹¹⁸

37While an industry-wide accepted definition of model evaluation remains elusive, and those that have been formulated are similarly broad to that found in the Code of Practice,¹¹⁹ technical literature may provide supplementary insight into how such evaluations are typically categorised. For instance, the Frontier Model Forum (“FMF”), an industry-supported organisation whose members include providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that are both signatories to the Code of Practice or otherwise subject to the obligations under Article 55,¹²⁰ identifies benchmark evaluations, red-team exercises, and controlled studies as the three main categories that capture most existing evaluation tasks.¹²¹ The evaluation techniques listed under Measure 3.2 can broadly be captured under these three categories.¹²² Model evaluations can also be distinguished by their focus, such as capability evaluations and propensity evaluations,¹²³ where capability evaluations measure whether a model has certain dangerous capabilities, while propensity evaluations capture whether the model has the propensity to harmfully apply those capabilities.¹²⁴ 

38Without being exhaustive,¹²⁵ the preceding paragraphs already illustrate the breadth and fragmentation of the evaluation landscape. This diversity could create practical challenges for both providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. and regulators when determining what constitutes state-of-the-art model evaluations under Article 55. In practice, model evaluations are conducted using a combination of publicly available benchmarks and bespoke, in-house evaluation methods.¹²⁶ Evaluations may be developed and run internally by providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , while others are outsourced to specialised third parties with controlled access to the model.¹²⁷ As a result, some of the most advanced evaluation techniques could remain proprietary and therefore not generally available on the market,¹²⁸ making it difficult for other providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ensure that their own model evaluation practices reflect the state of the art.¹²⁹

39The Safety and Security Chapter of the Code of Practice offers guidance to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. in determining what qualifies as state-of-the-art model evaluations, as well as how such evaluations may be conducted in a manner that demonstrates compliance with Article 55(1)(a).¹³⁰ It further clarifies that model evaluations are ‘integral along the entire model lifecycle.’¹³¹ At the stage of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification, model evaluation techniques, such as red-teaming, may be used to reveal unexpected capabilities or limitations of a model and thereby uncover risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. inherent to it.¹³² At the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment stage, adversarial evaluation techniques draw upon and feed into risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. modelling¹³³ to the extent that it requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to foresee and account for how malicious actors could circumvent model safeguards.¹³⁴ The results of these evaluations help determine which safety and security mitigations are appropriate.¹³⁵ Model evaluation techniques may also be used to test the effectiveness of those mitigations, for example by assessing whether safety mitigations successfully prevent the model from exhibiting previously identified harmful behaviours.¹³⁶ 

40In addition to the state-of-the-art model evaluations, the Code of Practice requires that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. conduct lighter-touch evaluations at appropriate trigger points along the entire model’s lifecycle.¹³⁷ These lighter-touch evaluations do not have to comply with the requirements set out in its Appendix 3, which provides guidance on the methodology of model evaluations,¹³⁸ but must still be appropriate to the purpose of assessing and mitigating systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . For example, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may conduct automated evaluations, which are tests run programmatically without the involvement of real users.¹³⁹ In line with the obligation for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to take appropriate measures throughout the model’s lifecycle and to cooperate with relevant actors along the AI value chain,¹⁴⁰ the Code of Practice thus requires continuous evaluation, albeit with varying levels of intensity.

2.1.1.2.1. State-of-the-art model evaluations

41Measure 3.2 in the Safety and Security Chapter reiterates that model evaluations must be ‘at least state-of-the-art’ and tailored to the modalities relevant to the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . The latter condition is not explicitly mentioned in the text of Article 55(1)(a) but aims to ensure that the evaluation meaningfully tests the conditions under which the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. could materialise.¹⁴¹ The Code of Practice’s Glossary in turn defines state of the art as ‘the forefront of relevant research, governance, and technology that goes beyond best practice.’¹⁴² Whether this definition should also be understood as reflecting the meaning of the state-of-the-art condition in Article 55(1)(a), or whether alternative interpretations remain possible, is discussed above.¹⁴³

42 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must design and conduct model evaluations that are not only state of the art but also appropriate to the model and the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. concerned.¹⁴⁴ The selection and design of suitable evaluation techniques should be informed by the model-independent information gathered pursuant to Measure 3.1, including insights into evaluation practices adopted by other providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. and the broader research community. In practice, this may require the development of risk-specific evaluation techniques tailored to particular systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. categories.¹⁴⁵ Model evaluations must be designed to capture all identified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and, at a minimum, the specified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. listed in Appendix 1.4.¹⁴⁶ For certain systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. categories – particularly chemical, biological, radiological, and nuclear (“CBRN”) and cyber-offence risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. – a relatively developed body of evaluation methods already exists.¹⁴⁷ Other risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. or capabilities, such as harmful manipulation or the capability to operate autonomously, remain comparatively under-evaluated.¹⁴⁸ This may require providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to design model evaluations from scratch, which in turn requires the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to assess not only the results of those evaluations but also whether the design of the evaluation itself is appropriate for assessing and mitigating the relevant identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

43Appendix 3 of the Safety and Security Chapter provides further guidance on the methodology of model evaluations.¹⁴⁹ First, model evaluations must meet the quality standard of having a high degree of scientific and technical rigour,¹⁵⁰ which in turn means that evaluations must have internal and external validity as well as be reproducible.¹⁵¹ Internal validity refers to the extent to which an evaluation ensures that the results of the evaluation ‘are as accurate as scientifically possible in the evaluation setting and are free from methodological shortcomings that could undermine the results.’¹⁵² The AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. may be able to verify the internal validity of model evaluations on the basis of the information provided in the Safety and Security Model Report.¹⁵³ External validity concerns the extent to which model evaluations are ‘suitably calibrated for results to be used as a proxy for model behaviour outside the evaluation environment.’¹⁵⁴ Developments such as changes in the deployment context of the AI model,¹⁵⁵ the emergence of new misuse techniques, or advances in evaluation methodologies may undermine the external validity of previously conducted model evaluations and may therefore prompt providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to update their Safety and Security Model Reports accordingly.¹⁵⁶ Finally, reproducibility requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to document the data, techniques, evaluation conditions, and other elements of the evaluation methodology in a manner that allows third parties, such as researchers and engineers, to validate, reproduce, or improve upon the results of the model evaluation.¹⁵⁷

44In addition to the model evaluations being rigorous, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must also ensure that model evaluations are conducted with at least a state-of-the-art level of model elicitation,¹⁵⁸ which refers to ‘technical work to systematically enhance a model’s capabilities, propensities, affordances, and/or effects, thereby facilitating an accurate measurement of the full range of its capabilities, propensities, affordances, and/or effects that can likely be attained’.¹⁵⁹ In particular, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are expected to employ techniques that minimise the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of under-elicitation – that is, situations in which the evaluation setup fails to reveal relevant capabilities – as well as model deception during evaluation, such as through sandbagging.¹⁶⁰ Under-elicitation risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. not being able to capture ‘an accurate measurement of the full range of its capabilities, propensities, affordances, and/or effects that can likely be attained.’¹⁶¹

45This requirement is particularly relevant in the context of adversarial testing, where red-teaming exercises must be sufficiently probing to reveal capabilities that might otherwise remain hidden and therefore escape meaningful risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification.¹⁶² Determining the appropriate level of model elicitation requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to take into account what is reasonably foreseeable in terms of potential misuse scenarios, the capabilities of likely misuse actors, and the expected deployment context of the model.¹⁶³

2.1.1.2.2. Adversarial testing of the model

46 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are expected to conduct adversarial testing of the model as part of the set of state-of-the-art model evaluations required under Article 55(1)(a). Adversarial testing refers to techniques of deliberately trying to subvert a model’s built-in defences by simulating hostile or manipulative interactions in which the tester assumes the role of an adversary¹⁶⁴ in order to assess whether the model can be induced to produce harmful or otherwise unacceptable outputs.¹⁶⁵ During this exercise, the adversarial tester can ‘identify dangerous capabilities, vulnerabilities, or other emergent properties’ of the model that might not be apparent under standard evaluation conditions.¹⁶⁶ 

47The AI Act identifies red-teaming as one form of adversarial testing that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must conduct within the suite of state-of-the-art model evaluations for assessing systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .¹⁶⁷ Red-teaming is an expert-driven and scenario-focused exercise that evolved in the cybersecurity defence sector¹⁶⁸ where experts use various tools and techniques to emulate how an adversary would attempt to identify and exploit system vulnerabilities.¹⁶⁹ AI developers have increasingly adopted red-teaming techniques to identify risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. and assess the robustness of safety and security mitigations in AI models and systems.¹⁷⁰ Unlike traditional cybersecurity red-teaming,¹⁷¹ AI red-teaming practices go beyond identifying security flaws and instead probe how a model can be induced to generate ‘harmful, unwanted, or policy-violating outputs’¹⁷², with the aim of assessing and managing ‘the safety, security, and trustworthiness of these models’¹⁷³.

48Red-teaming raises a structural question concerning the delineation between evaluating a model and evaluating a system.¹⁷⁴ The AI Act draws a conceptual distinction between (general-purpose) AI models and AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. built on top of them.¹⁷⁵ However, certain systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may only become apparent once a model is integrated into downstream systems, particularly where such systems provide access to tools or other forms of operational scaffolding.¹⁷⁶ The Code of Practice therefore requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , when designing and conducting red-teaming exercises, to account within the limits of reasonable foreseeability for both the elicitation capabilities of potential misuse actors and the expected use context of the model.¹⁷⁷ This includes considering planned or contemplated integrations of the model into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , as well as integrations currently observed for similar models where such uses are known to the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. and cannot reasonably be excluded for their own model.¹⁷⁸

2.1.1.2.3. Internal and external model evaluations

49Closely related questions also arise concerning the extent to which Article 55(1)(a) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to conduct independent external adversarial testing or other types of external model evaluations.¹⁷⁹ External model evaluations are understood within the industry as encompassing evaluations conducted by, or with the involvement of, independent external actors.¹⁸⁰ 

50The AI Act distinguishes between internal and external model evaluations in Recital 114, which states that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should conduct the necessary model evaluations, including, ‘as appropriate, through internal or independent external testing.’ Where applicable, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must then document any measures put in place ‘for the purpose of conducting internal and/or external adversarial testing’.¹⁸¹ This distinction reflects meaningful technical and policy differences between the two types of model evaluations. External model evaluations can enhance trust in the results, while also carrying commercial, security, and operational disadvantages that internal model evaluations do not.¹⁸² 

51External independent assessments are commonly mandated across EU product-safety and safety-critical legislation,¹⁸³ wherein such obligations are imposed explicitly as freestanding requirements, most notably through notified-body conformity-assessment procedures or explicit audit obligations.¹⁸⁴ The omission of a reference to internal and external testing in the text of Article 55(1)(a) thus leaves the scope of this obligation open to interpretative ambiguity.

52One interpretive pathway is to read the obligation to conduct state-of-the-art model evaluations as, in certain circumstances, implicitly requiring both internal and external model evaluations. While Article 55(1)(a) itself does not expressly refer to external evaluations, Recital 114 clarifies that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are to conduct model evaluations, including, ‘as appropriate, through internal or independent external testing’, as mentioned above. Read together with the state-of-the-art requirement, the recital suggests that whether external evaluations are required depends on whether internal evaluations alone are sufficient to meet the level of rigour necessary for adequately assessing and mitigating systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Put differently, where certain systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can only be assessed through independent external evaluation, reliance exclusively on internal evaluations may fall short of the state-of-the-art standard required under Article 55(1)(a). At the same time, the ‘as appropriate’ language in Recital 114 signals that the requirement to conduct external model evaluations may be conditional rather than automatic.

53The Safety and Security Chapter of the GPAI Code of Practice, as the European Commission’s endorsed operationalisation of Article 55(1),¹⁸⁵ lends support to this reading. According to Measure 3.2 and the accompanying Appendix 3.5, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are expected to conduct independent external model evaluations in addition to internal model evaluations prior to market placement, unless the model can be demonstrated to be ‘similarly safe or safer’ than existing models or where signatories are unable to ‘appoint adequately qualified independent external evaluators’.¹⁸⁶ By making external model evaluations conditional in this way, the Code of Practice appears to reflect an attempt to balance two competing considerations: on the one hand, a purposive reading of Article 55 directed at ensuring appropriate assessment and mitigation of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , and on the other, the need to maintain proportionality in the compliance measures imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .¹⁸⁷

54A second interpretative pathway is that the omission of external model evaluations from the text of Article 55(1)(a) is deliberate, with the effect that such evaluations fall outside the scope of the provision. As noted above, the only reference to external model evaluations appears in Recital 114, which may provide interpretative guidance for the operative provision but cannot introduce additional substantive obligations into the text of that provision.¹⁸⁸ This is particularly relevant if conducting external model evaluations is understood to be a materially different obligation from conducting internal model evaluations. The second and only other reference to external model evaluations in the AI Act appears in Annex XI, which is an integral and binding part of the AI Act. However, its reference to external model evaluations is framed as part of the technical documentation to be provided ‘where applicable’.¹⁸⁹ This wording suggests that external evaluations are contemplated by the Act and may be relevant in some circumstances, but does not necessarily support the conclusion that Article 55(1)(a) requires all providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to conduct external evaluations as a matter of course. On that reading, interpreting Article 55(1)(a) as implicitly requiring external model evaluations would extend beyond what is supported by the text of the operative provision itself.

55The tenability of this line of interpretation, however, is not assured. Its persuasiveness depends in part on establishing that requiring external model evaluations would be a disproportionately burdensome reading of the obligation to perform model evaluations. This can be contested in light of the overall risk-based approach and safety objectives of the AI Act.¹⁹⁰ A purposive interpretation of Article 55(1)(a), directed at ensuring effective assessment and mitigation of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ,¹⁹¹ may support the conclusion that external model evaluations are necessary for appropriate risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management, given their ‘unique benefits [such as] broader researcher participation, diversity of subject matter experts, novel approaches, independence, and greater evaluation speed’.¹⁹² The more pertinent question may therefore not be whether external model evaluations fall within the provision’s scope at all but rather under which conditions they are appropriate. In that respect, factors such as the absence of prior independent external evaluations, limitations in the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s internal evaluation capacity, or the inability of internal evaluations alone to demonstrate sufficient robustness or independence may support the conclusion that external evaluations are ‘appropriate’ within the meaning of Recital 114.

56The GPAI Code of Practice provides guidance not only on when external model evaluations are appropriate, but also on how they should be conducted.¹⁹³ It sets out criteria for identifying adequately qualified independent external evaluators and specifies what constitutes adequate access to the model in the course of an external evaluation.¹⁹⁴ Significantly, the Safety and Security Chapter defines an independent external evaluator as any ‘natural or legal person that has no financial, operational, or management dependence on the Signatory […] and is otherwise free from the Signatory’s control in reaching conclusions and/or making recommendations, including through contractual safeguards and suitable conflict of interest policies.’¹⁹⁵

2.1.13. Documentation obligations under Article 55(1)(a) and (b)

57The text of Article 55(1)(a) explicitly refers to documentation only in a specific context, namely by requiring providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to perform and document adversarial testing. This wording gives rise to the next interpretative question on whether documenting the state-of-the-art model evaluations is also a constitutive element of this provision. Legal scholarship appears to be divided between authors who consider documentation exclusively in relation to adversarial testing¹⁹⁶ and those who understand compliance with Article 55(1)(a) as requiring both the execution of the model evaluation and its documentation, where the latter includes the results, the tests performed and the criteria applied.¹⁹⁷ 

58Treating documentation as a distinct obligation built into Article 55(1)(a) can be substantiated on the basis of a teleological and systematic interpretation of the provision. From a teleological perspective, an obligation limited to conducting model evaluations without a corresponding requirement to document them would render the duty to identify and mitigate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. having regard to the state of the art largely unenforceable. In the absence of documentation, the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. would be unable to verify either the methodologies relied upon or their adequacy in light of the state-of-the-art requirement. A teleological reading of Article 55(1)(a) thus entails that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must not only conduct model evaluations but also document them in a manner that enables effective supervisory scrutiny. Interpretation to the contrary would undermine the regulatory purpose of Article 55(1)(a), which is to subject Article 3(58) AI Act: ‘subject’, for the purpose of real-world testing, means a natural person who participates in testing in real-world conditions. models posing increased systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to enhanced oversight in order to safeguard public goods.¹⁹⁸

59A systematic reading of Article 55(1) in conjunction with Annex XI Section 2. speaks to the same effect.¹⁹⁹ Notably, the absence of an explicit cross-reference between Article 55(1) and Annex XI, which in itself is a departure from standard guidance on the drafting of EU regulations,²⁰⁰ cannot be relied upon to negate the normative relationship between the two, given that annexes form an integral part of EU legislative acts and have the same binding legal status as the operative provisions.²⁰¹ Indeed, while Article 53 expressly links the technical documentation obligation to Annex XI,²⁰² Article 55 does not contain an equivalent internal reference, and the title of Annex XI refers only to Article 53. However, the title of Section 2. of Annex XI leaves little room for ambiguity as to its function in supplementing and operationalising the obligations laid down in Article 55.

2.1.1.3.1. The scope of information to be compiled under Article 55(1)(a) and (b)

60The scope of the documentation obligation under Article 55(1)(a) and (b) is specified through Annex XI Section 2., which sets out the categories of information that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may be required to supply to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. .²⁰³ This includes technical documentation containing, inter alia, a detailed description of model evaluation strategies, evaluation results, and methodologies.²⁰⁴ Annex XI also requires, where applicable, a description of the measures put in place for conducting internal or external adversarial testing, as well as a description of the model’s system architecture.²⁰⁵

61 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who are also signatories to the GPAI Code of Practice are required to compile and keep up to date a Safety and Security Model Report containing detailed information on their systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation processes to be shared with the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. before placing a model on the market.²⁰⁶ For providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that are signatories to the Code of Practice, it is conceivable that the Commission may rely primarily on the Safety and Security Model Report when requesting information,²⁰⁷ rather than also requesting the provision of the information compiled under Annex XI, insofar as the information required by the Annex is already covered by the Safety and Security Model Report.²⁰⁸ For providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that are not signatories to the Code of Practice, requests for information compiled under Annex XI Section 2. are likely to be informed by the structure and scope of information reflected in the Safety and Security Model Report, given its role as an operational benchmark for demonstrating compliance with Article 55.²⁰⁹ 

2.1.1.3.2. The extent of required documentation

62Article 55(1)(a) says that state-of-the-art model evaluations must be conducted and documented²¹⁰ with a view of identifying and mitigating systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²¹¹ One interpretative question that arises in this context relates to the extent and detail of documentation required.

63The Safety and Security Chapter of the GPAI Code of Practice emphasises that the level of detail in documentation and reporting should be proportionate to the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²¹² Having an appropriate level of detail in the supplied documentation is therefore important for demonstrating the technical rigour of model evaluations.²¹³ Subpar documentation practices, or evidence that other providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. have identified and implemented more rigorous approaches to documentation, could render a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s own documentation practices inadequate for the purposes of demonstrating compliance.²¹⁴ This, in turn, would prompt providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to adjust their documentation practices accordingly. In practice, the documentation landscape remains fragmented, although there are emerging efforts to converge towards higher standards.²¹⁵ Moreover, there is arguably a spillover from the state-of-the-art requirement as extending to documentation practices. Given how fundamental documentation is to the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process, this could contribute to a similar upward dynamic where providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are incentivised to invest greater technical effort and innovation into ‘[advancing] the state of the art in AI safety and security and related processes’.²¹⁶

64The documentation compiled by providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must, at a minimum, be sufficiently detailed to enable the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to assess whether the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. has adequately assessed and mitigated systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²¹⁷ Documentation therefore performs an evidentiary function, in the sense that it must render the evaluation process, its methodology, and its results legible to the EU regulator.²¹⁸ At the same time, the breadth and depth of documentation are constrained by considerations of proportionality.²¹⁹ The AI Act does not require providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to document their processes in a manner that would impose disproportionate burdens, nor to disclose more information than is necessary for assessing compliance.²²⁰ This creates a tension between, on the one hand, ensuring that the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. can meaningfully evaluate the adequacy of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation, and, on the other, avoiding excessive documentation requirements.

2.1.2. Article 55(1)(b): Assessment and mitigation of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level

65Article 55(1)(b) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ‘assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level, including their sources, that may stem from the development, the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , or the use of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .’²²¹ The following analysis will tackle several key interpretative questions pertinent to the interpretation of this obligation. It begins by examining the type of possible  risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that are subject to the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process. It then outlines the broader risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation framework that spans Article 55(1)(a), (b) and (d),²²² including how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. can demonstrate compliance along each step of this iterative process. In carrying out these obligations, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to adopt and implement appropriate measures to ensure that systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are reduced to an acceptable level. The meaning of appropriate and acceptable is examined in Section 2.1.2.3. Finally, the analysis considers at which stages of the AI model lifecycle the AI Act requires risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures.

2.1.2.1. ‘Possible’ systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level

66Article 55(1) AI Act limits risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation to possible  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that have an effect at the Union level and that ‘may stem from the development, the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , or the use of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’.²²³ The explicit reference to systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. having an effect at the Union level is somewhat curious, given that having ‘a significant impact on the Union market’ already constitutes an essential characteristic of the definition of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²²⁴ Indeed, for risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to qualify as systemic under Article 3(65), and for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to qualify as providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. under Article 51, the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. they present must already have a significant impact on the Union market so as to justify EU intervention on the basis of Article 114 TFEU.²²⁵

67The possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to be identified are those that ‘may stem from the development, the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , or the use’ of the GPAI model.²²⁶ The term ‘may’ signals that, while systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can emerge at various stages – development, placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , and use – this formulation does not necessarily exclude the possibility that such risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. arise elsewhere along the AI model’s lifecycle,²²⁷ which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to monitor as part of their ongoing risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management obligations.²²⁸ In practice, this has been described as imposing a potentially ‘boundless’ obligation, given that, by their very nature as being general-purpose, GPAI models may be used across a wide range of contexts and must therefore be monitored in a very broad set of scenarios, thereby potentially challenging legal certainty.²²⁹ Section 2.1.2.4. below examines how the obligations of assessment and mitigation span the entire model lifecycle, from development through to downstream use.

68The question of what amounts to a possible  systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is closely linked to the level of effort that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are expected to invest in anticipating risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. at the identification stage. Notably, the AI Act itself does not systematically employ the term possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.  in other contexts, which leaves its precise scope open to interpretation. The GPAI Code of Practice differentiates between specified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , which are those listed in Appendix 1.4, and all other potential systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.  that ‘could stem from the model and be systemic’.²³⁰ The notion of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.  most plausibly encompasses both the specified risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identified in the Code of Practice and the broader category of potential systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. envisaged therein.

69A useful point of comparison is the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management framework applicable to high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 9, which requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to identify ‘known and reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’ that may stem from the intended use or reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. of the system.²³¹ This formulation anchors the obligation in a standard of foreseeability, even if the AI Act does not further specify how that standard is to be operationalised.²³² By contrast, Article 55 refers to ‘possible’ systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , raising the question of whether this notion extends beyond reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .

70Possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. should, at a minimum, be understood as encompassing both known and reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. . Known risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. are those that refer to ‘harm [that] has occurred in the past or is certain to occur in the future.’²³³  Risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. can become known not just by virtue of the subjective knowledge of a specific provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , but also if these risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. have been subject to significant media attention or entered into a recognised incident database and thus can be assumed to be known, even if the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. does not use the database in question or has overlooked the specific entry.²³⁴ 

71The key interpretative question is whether the notion of possible  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. extends beyond the standard of reasonable foreseeability. While the use of the term possible could signal a deliberate departure from the wording of Article 9 and be read as encompassing more remote or even speculative risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , such an interpretation may be difficult to reconcile with the principles of proportionality and legal certainty.²³⁵ Requiring providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to account for purely hypothetical or unforeseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. would risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. rendering the obligation effectively unbounded.²³⁶ At the same time, given the potentially more severe and wide-ranging impacts of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may be expected to adopt a more rigorous and forward-looking approach to identifying and analysing risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. . Indeed, where the potential harm is severe or even catastrophic, the gravity of the consequences may outweigh a low probability of occurrence.²³⁷ Accordingly, while the standard of reasonable foreseeability remains the appropriate baseline, its application in this context is likely to require a more precautionary assessment than under Article 9.²³⁸

2.1.2.1.1. Possible systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. corresponds to reasonably foreseeable risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm.

72There are several arguments that support reading possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to correspond to reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. . First, the challenge of setting boundaries for foreseeability is fundamentally one of legal certainty: providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must be able to determine when ‘they are allowed to stop looking for new risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’.²³⁹ Framing the notion of possible  systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. in terms of a more functionally useful legal concept of reasonable foreseeability could help to address this concern.²⁴⁰ Second, the principle of proportionality, both as a guiding principle for the interpretation of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management obligations, and as a motivation behind the EU legislature’s drafting choices, supports the view that the identification of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must remain constrained by a standard of reasonable foreseeability.²⁴¹ The European Parliament and Council amended the Commission’s version of Article 9 to include the notion of reasonableness ‘to keep the burden of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management in proportion to risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .’²⁴² A third argument in favour of this interpretation draws on considerations of systematic consistency across the AI Act. The risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process set out in Article 9 and the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process in Article 55(1) share the same definition of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm.  in Article 3(2), are directed at protecting the same public interests – including public health, safety, and fundamental rights – and draw on established risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management frameworks reflected in the ISO/IEC guidelines.²⁴³ Indeed, historical versions of the negotiated AI Act proposal reveal that ‘the Council has suggested extending Article 9 to “general purpose AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ”’,²⁴⁴ at a time when the term general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market.  had not yet been introduced.²⁴⁵ These shared structural features support a consistent interpretation of the amount of effort providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. need to invest in identifying and analysing reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .

73Based on the above, the way in which reasonable foreseeability is interpreted and applied for the purposes of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification and analysis under Article 9(2)(a) can inform how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identify and analyse possible  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. under Article 55(1). A risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. is foreseeable if it ‘has not yet occurred but can already be identified.’²⁴⁶ Reasonableness has been interpreted as an objective standard that allows for the strengths of a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , such as their ‘specific knowledge [to] individually tighten the standard of care for foreseeability [while] subjective grounds […] such as lack of knowledge, insufficient training, or lack of experience remain irrelevant.’²⁴⁷ For providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the standard of reasonableness would be calibrated in reference to factors including their expertise, the foreseeability of the damage, ‘the availability and the costs of precautionary or alternative methods,’ and ‘the nature and value of the protected interest involved,’²⁴⁸ which, in this case, includes risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to public health, safety, public security, fundamental rights, or the society as a whole.²⁴⁹ 

2.1.2.1.2. Reasonably foreseeable systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

74While the notion of possible may thus correspond to reasonably foreseeable, the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process under Article 55(1) may be read as requiring providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to undertake more extensive efforts in identifying and analysing risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. than those imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .

75It is possible to justify this interpretation by reference to the nature and characteristics of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ,²⁵⁰ as well as the ‘potential significantly negative effects’ that warrant such models being subject to the relevant obligations under the AI Act,²⁵¹ even where exemptions would otherwise apply to GPAI models without systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²⁵² The GPAI Code of Practice confirms that systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , as defined in Article 3(65), should be interpreted in light of both the probability and severity of harm reflected in the definition of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm.  in Article 3(2),²⁵³ while also taking account of additional characteristics that confer its systemic nature. These include compounding or cascading effects, high velocity, and the fact that a small number of actors or events could trigger the materialisation of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , which may in turn be difficult or impossible to reverse.²⁵⁴ Given the ways in which systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may materialise, it is reasonable to argue that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. should be required to assess and mitigate not only reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , but also a broader category of possible  risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , subject to a more stringent standard of effort rather than the reasonableness threshold providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. are expected to adhere to. As has been noted elsewhere, ‘the greater the potential impact of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , the more effort an organisation needs to put into foreseeing it. […] it should be extremely difficult for a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to credibly assure that a catastrophic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. was unforeseeable’.²⁵⁵ 

76Evidence supporting this interpretation can be drawn from the relationship between reasonable foreseeability and the state-of-the-art requirement, both of which function as standards that structure the scope and intensity of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’ obligations. In the context of Article 9, commentators have argued that, in assessing both known and reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , the objective standard of care must also take account of the generally acknowledged state of the art. This implies that neither purely theoretical nor entirely improbable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. are covered.²⁵⁶ Rather, the effort required to identify and assess reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. must be calibrated to what is generally acknowledged as the state of the art, which in practice corresponds to established best practices.²⁵⁷ 

77Article 55(1) similarly requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to employ methods and tools that reflect the state of the art for the purposes of assessing and mitigating systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . As discussed in Section 2.1.1.1. above, the notion of state of the art mentioned in Article 55(1)(a) arguably sits at a higher threshold than the generally acknowledged state of the art used elsewhere in the AI Act as it requires measures that go beyond established best practices.²⁵⁸ Therefore, where the assessment and mitigation of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must reflect this higher standard, limiting possible  risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. may not be sufficient if the state of the art demands a greater degree of effort from providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .

78This reading appears to be endorsed by the Code of Practice’s Safety and Security Chapter’s Recital (g) on the precautionary principle.²⁵⁹ The recital recognises that the lack and subpar quality of data surrounding systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may impede the assessment of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and instead compels providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to extrapolate from current adoption rates and research and development trajectories of models when identifying systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . This forward-looking approach to risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification is also reflected in Measure 2.2 of the Code of Practice, which requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to develop appropriate  systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenarios as a basis for future risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. modelling.²⁶⁰ This exercise necessarily involves specifying ways in which systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from a model might materialise.²⁶¹ For the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenarios to be appropriate, the providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will have to rely on ‘best practices, the state of the art, or other more innovative processes, measures, methodologies, methods, or techniques that go beyond the state of the art.’²⁶² As a result, where providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. deploy appropriate measures that go beyond the state of the art, they may become able to and also enable other providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to identify and analyse systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that extend beyond what is merely reasonably foreseeable.²⁶³ The state-of-the-art condition effectively lowers the threshold of foreseeability,²⁶⁴ as advances in methods and techniques make it possible to capture risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that would previously have fallen outside the scope of reasonable foreseeability.

79Further support for interpreting possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as extending beyond reasonable foreseeability can be drawn from the structure of Article 9 and the limiting clauses embedded in that provision. Article 9 limits the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management system to only those risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that can be reasonably mitigated or eliminated through the development or design of the high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , or through the provision of adequate technical information.²⁶⁵ The absence of a comparable limiting clause in Article 55(1) suggests a broader range of possible risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. subject to assessment and mitigation. To this point, the safety mitigations listed in the Code of Practice’s Safety and Security Chapter are not confined to the design and development stage²⁶⁶ but extend into deployment,²⁶⁷ governance,²⁶⁸ and building a safe ecosystem around the model.²⁶⁹ 

80In addition, Article 9 limits identification and analysis of risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to those arising from the use of high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. in accordance with its intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. and under conditions of reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. .²⁷⁰ By contrast, GPAI models are characterised by their significant generality and by the fact that they may form the basis for a wide range of downstream systems, uses and applications.²⁷¹ Combined with the potentially greater severity of harm and the broader range of public interests at stake,²⁷² this structural difference suggests that the assessment and mitigation of possible  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. under Article 55 cannot be confined to what is reasonably foreseeable within the narrower risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process under Article 9.

2.1.2.1.3. Sources of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

81 Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is defined as being specific to high-impact capabilities Article 3(64) AI Act: ‘high-impact capabilities’ means capabilities that match or exceed the capabilities recorded in the most advanced general-purpose AI models. and as increasing ‘with model capabilities and model reach, [and] can arise along the entire lifecycle of the model’.²⁷³ The wording ‘specific to’ has been discussed as meaning that systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stems from high-impact capabilities Article 3(64) AI Act: ‘high-impact capabilities’ means capabilities that match or exceed the capabilities recorded in the most advanced general-purpose AI models. , or, in other words, that high-impact capabilities Article 3(64) AI Act: ‘high-impact capabilities’ means capabilities that match or exceed the capabilities recorded in the most advanced general-purpose AI models. are the main source of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²⁷⁴ It has also been posited that, on an alternative interpretation, systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is identified in models that have high-impact capabilities Article 3(64) AI Act: ‘high-impact capabilities’ means capabilities that match or exceed the capabilities recorded in the most advanced general-purpose AI models. but does not stem exclusively from them.²⁷⁵ Instead, systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may arise in the most advanced GPAI models while also being shaped by additional factors, including ‘conditions of misuse, model reliability, model fairness and model security, the level of autonomy of the model, its access to tools, novel or combined modalities, release and distribution strategies, the potential to remove guardrails and other factors.’²⁷⁶ 

82In identifying the sources of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should therefore take account not only of factors internal to the model, such as its capabilities, but also of factors and conditions external to the model. This reading finds support in Recital 110 and is, more importantly, expressly reiterated in Appendix 1.3 of the Code of Practice’s Safety and Security Chapter. The latter enumerates selected model capabilities,²⁷⁷ model propensities,²⁷⁸ model affordances,²⁷⁹ and contextual factors as part of a non-exhaustive list of potential sources of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. for the purposes of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification.

2.1.2.2. Assessment and mitigation of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

83 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are required to assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .²⁸⁰ The terms assessment and mitigation are not defined in the AI Act. To this end, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may rely on the GPAI Code of Practice for guidance on how to implement the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process. In particular, the Safety and Security Chapter describes risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment as encompassing the steps of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification,²⁸¹  systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis,²⁸² and systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determination.²⁸³  Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.  mitigation requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to implement both safety and security measures.²⁸⁴ The full systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation process is continuous and iterative,²⁸⁵ spanning the model’s entire lifecycle,²⁸⁶ and updated until systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is brought to an acceptable level.²⁸⁷ 

84The Code of Practice’s structuring of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process is also consistent with the way in which the AI Act envisions risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .²⁸⁸ Article 9(2) explicitly lists risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification, risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. analysis, and risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. evaluation as distinct and sequential steps within a ‘continuous iterative process planned and run throughout the entire lifecycle of a high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’.²⁸⁹ Arguments can thus be levied in support of a systematic reading of the AI Act,²⁹⁰ under which the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process is understood to follow a similar structure across providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

85Interpretative support may also be drawn from international risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management standards, in particular ISO/IEC Guide 51 and the related standards ISO 31000 and ISO 73.²⁹¹ While these standards are not expressly referenced in the AI Act, the terminology and risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management structure adopted in the Code of Practice (and in Article 9)²⁹² mirrors their risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management framework.²⁹³ The following analysis therefore draws on the ISO standards only insofar as their terminology and structure support the interpretation of Article 55(1) and the corresponding commitments in the GPAI Code of Practice.²⁹⁴ Where the term assessment is used throughout the rest of this chapter, it should thus be understood as encompassing the identification, analysis, and acceptance determination of possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

2.1.2.2.1. Risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment 2.1.2.2.1.1. Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification

86 Risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification is the starting point of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process,²⁹⁵ at which stage providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. engage in ‘finding, recognising and describing’ all possible risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. stemming from the model.²⁹⁶ The AI Act does not provide specific methods for identifying risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. . Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may therefore draw on established risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification techniques and methodologies,²⁹⁷ including those described in the GPAI Code of Practice.

87Under the Code of Practice’s Safety and Security Chapter, systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification proceeds in two steps. First, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. need to follow a structured process to identify and compile a list of potential²⁹⁸  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identified through a broader information-gathering exercise about the model,²⁹⁹ and the specified  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that have been pre-identified and listed in Appendix 1.4.³⁰⁰

88For the purposes of identifying potential systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , signatories are expected to draw on a set of five ‘distinct but in some cases overlapping types of risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’ listed in Appendix 1.1. These include risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to public health, safety, public security, fundamental rights, and society as a whole. This categorisation reflects a broad set of protected public interests and largely mirrors the definition of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. set out in Article 3(65).³⁰¹ Appendix 1.1 further provides a non-exhaustive list of examples falling within these categories, including risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of major accidents; risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. affecting critical sectors or infrastructure; impacts on public mental health; and risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to fundamental rights such as freedom of expression and information, non-discrimination, privacy, and the protection of personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. . It also includes risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to the environment, non-human welfare, economic security, and democratic processes, as well as risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. arising from the concentration of power and from illegal, violent, hateful, radicalising, or false content, including child sexual abuse material (“CSAM”) and non-consensual intimate images (“NCII”).

89Based on the types of risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. listed in Appendix 1.1, signatories are required to consider a range of information sources, including (i) model-independent information; (ii) relevant information concerning the model and similar models, including information derived from post-market monitoring, serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , and near misses; and (iii) any other relevant information communicated to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. , the Scientific Panel, or other relevant initiatives.³⁰²  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are then required to analyse relevant characteristics of the compiled risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ,³⁰³ such as their nature and sources,³⁰⁴ and to identify which of them constitute systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model.³⁰⁵

90Alongside potential systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are by default required to assess whether their model displays any of the specified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. listed in Annex 1.4 of the Code of Practice.³⁰⁶ These include risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. relating to (1) chemical, biological, radiological and nuclear harms; (2) loss of control; (3) cyber-offence; and (4) harmful manipulation.³⁰⁷ The specified  systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are based on and may correspond to the types of risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. listed in Annex 1.1,³⁰⁸ which in turn include risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. listed in Recital 110.

91For each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are then required to develop appropriate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenarios, including by determining the number of such scenarios and the level of detail at which they are described.³⁰⁹ The Glossary defines a systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenario as a scenario in which a systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from a model might materialise.³¹⁰ These systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenarios are to form the basis for the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. modelling that signatories must undertake as part of the subsequent systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis stage of the full assessment and mitigation process.³¹¹

2.1.2.2.1.2. Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis

92Signatories are required to engage in risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. analysis to develop as complete an understanding as possible of the identified risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , including their nature, sources, and level.³¹² In situations of high uncertainty, the ISO 31000 guidelines recommend using a combination of qualitative and quantitative techniques to assess the probability of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. occurring and the magnitude or level of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. should it materialise.³¹³ Any determinations as to the level of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. made at this stage will inform the next step of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determinations. Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may, and signatories should, rely on the Code of Practice for guidance on what methods and techniques to use for the purposes of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis.

93More specifically, Commitment 3 of the Safety and Security Chapter requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to analyse each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , with the outcome of that analysis informing the subsequent determination of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance.³¹⁴ At this stage, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must first gather model-independent information relevant to the identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. using methods such as web searches and literature reviews, market analyses that involve assessing the capabilities of other models, reviews of training data Article 3(29) AI Act: ‘training data’ means data used for training an AI system through fitting its learnable parameters. for indications of data poisoning or tampering, and analyses of historical incident data and incident databases.³¹⁵ The breadth and depth of this information-gathering exercise depend on the probability and severity of harm.³¹⁶  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. then need to conduct model evaluations that are at least state of the art and appropriate to both the model and the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. in question.³¹⁷ The third step requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to conduct systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. modelling,³¹⁸ which involves specifying the pathways through which a systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may materialise.³¹⁹ 

94 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should then use at least state-of-the-art risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. methods to estimate both the probability and the severity of harm for each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .³²⁰ Estimates of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. could be expressed using formats such as risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. scores, risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. matrices,³²¹ probability distributions, or other suitable representations, and may be quantitative, semi-quantitative, or qualitative in nature. The fifth and final element of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis process is post-market monitoring.³²² This process involves gathering information about the model’s capabilities, propensities, affordances, and/or effects over the period from when the model is placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. until the retirement of the model from being made available on the market Article 3(10) AI Act: ‘making available on the market’ means the supply of an AI system or a general-purpose AI model for distribution or use on the Union market in the course of a commercial activity, whether in return for payment or free of charge. .³²³ During post-market monitoring, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. commit to providing adequate access to the model to an adequate number of independent external evaluators.³²⁴  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are exempt from having to grant access to external evaluators where the model qualifies as a similarly safe or safer model with regard to the same systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , as specified in Appendix 2.2.³²⁵

2.1.2.2.1.3. Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determination

95 Risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. acceptance determination, which also corresponds to risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. evaluation in established risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management practices,³²⁶ is the next step in the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment process, at which it is necessary to determine whether the analysed risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. are acceptable or whether they exceed a level that cannot be tolerated and instead require mitigation.³²⁷ This is achieved by comparing the results of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. analysis against previously established risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. criteria and determining where risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. reduction measures are needed.³²⁸ 

96Commitment 4 of the Safety and Security Chapter covers systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determination. This commitment requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to specify systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance criteria, which are then used to determine whether each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , as well as the overall systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. profile, are acceptable.³²⁹ These criteria must incorporate a safety margin to account for potential limitations, uncertainties, or changes relating to the source of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment, and the effectiveness of the mitigation measures.³³⁰ The signatories have discretion to develop acceptance criteria suitable for the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at issue unless criteria are prescribed for specified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. pursuant to Appendix 1.4.³³¹ In evaluating the specified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as listed in Appendix 1.4, signatories commit to using appropriate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. tiers defined in terms of model capabilities, and may additionally incorporate model propensities, risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. estimates,³³² and other metrics.³³³ The tiers must be measurable and include at least one systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. tier that has not yet been reached.³³⁴

97Only where, on the basis of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance criteria, each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and the overall systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are determined to be acceptable may providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. proceed with the development, placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , and/or use of the GPAI model.³³⁵ Should the results of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determination reveal that the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model are unacceptable, or are reasonably foreseeable to soon no longer be determined acceptable, the providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must refrain from making the model available on the market, or, where necessary, restrict its availability, withdraw it, or recall it.³³⁶ Signatories are required to then implement appropriate safety and security mitigations pursuant to Commitments 5 and 6 respectively and conduct another round of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification, systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. analysis, and systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. acceptance determination until the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are deemed to be acceptable.³³⁷ 

2.1.2.2.2. Risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation

98At the stage of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to select and implement measures to address and mitigate identified systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .³³⁸  Risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation is an iterative process involving the selection and implementation of mitigation measures, the assessment of whether the resulting residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. has been brought to an acceptable level, and, where it has not, the identification and implementation of further measures.³³⁹ Given that the AI Act does not prescribe how risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation is to be conducted or what specific mitigation measures must be implemented for the purposes of complying with Article 55(1)(b), the primary responsibility lies with providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to identify and make the case for what they consider to be appropriate mitigations.³⁴⁰ At the same time, the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. retains the authority to assess the appropriateness of chosen mitigation measures and may, through the structured dialogue mechanism under Article 93(2), signal what it considers to constitute appropriate risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation.³⁴¹ The GPAI Code of Practice offers further guidance on how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may implement such measures for the purposes of demonstrating compliance with their obligations.³⁴² 

99The Safety and Security Chapter of the GPAI Code of Practice requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to implement appropriate safety and security mitigations along the model’s entire lifecycle.³⁴³ Security mitigations are framed in terms of ensuring an adequate level of cybersecurity protection for the model and its physical infrastructure, with a view to preventing systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that may arise from unauthorised access, release, and/or model theft.³⁴⁴ Security mitigations are particularly relevant to the obligation set out in Article 55(1)(d).³⁴⁵ Safety mitigations are directed at ensuring that the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model are brought to an acceptable level.³⁴⁶ Measure 5.1 of the Safety and Security Chapter contains a non-exhaustive list of safety mitigations providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may implement in the course of bringing systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to an acceptable level.³⁴⁷ This may include, for example, technical measures at the level of the AI model itself (such as filtering inputs or outputs), technical instructions to downstream providers Article 3(68) AI Act: ‘downstream provider’ means a provider of an AI system, including a general-purpose AI system, which integrates an AI model, regardless of whether the AI model is provided by themselves and vertically integrated or provided by another entity based on contractual relations. to provide certain information or notices, or monitoring and reviewing the model for risky behaviour.

100The requirement that mitigation measures be ‘appropriate’ implies that they must be sufficiently robust under adversarial conditions, taking into account, inter alia, the model’s release and distribution strategy, which may itself constitute a source of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .³⁴⁸ Model evaluations are particularly relevant at the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation stage,³⁴⁹ as they can be employed to test the effectiveness of the safety mitigations by revealing, for example, whether the model remains susceptible to jailbreaking or other adversarial attacks.³⁵⁰ The results of the model evaluations will determine whether improved mitigation measures must be implemented, or whether the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model are acceptable and providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may thus proceed with the development, making available on the market Article 3(10) AI Act: ‘making available on the market’ means the supply of an AI system or a general-purpose AI model for distribution or use on the Union market in the course of a commercial activity, whether in return for payment or free of charge. , or use of the model.³⁵¹ 

101Further interpretative guidance may also be drawn from the way in which the AI Act frames risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation in the context of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .³⁵² The risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process outlined in Article 9, similarly to the framework constructed around GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , imposes a similar sequence of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. identification, risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. analysis and risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management upon providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . Notably, unlike Article 9, Article 55 does not impose comparable limitations on the scope of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation measures that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are required to implement.³⁵³ Under Article 9(3), risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation is confined to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that can be adequately mitigated or eliminated through system design and development, or through the provision of appropriate technical information. In other words, the scope of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management under Article 9 is confined to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that can be addressed through the measures listed in Article 9(5)(a)–(c), with the consequence that where residual risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. cannot be reduced to an acceptable level the system may not be placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. .³⁵⁴ The absence of an equivalent limitation in Article 55(1)(b) suggests that the scope of mitigation for GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may indicate that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to engage with systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. more broadly, including in situations where such risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. cannot be fully mitigated through design, or technical measures, or provision of information alone.

2.1.2.3. ‘Appropriate’ measures for ‘acceptable’ risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm.

2.1.2.3.1. Appropriate risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures

102The objective of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation process is to ensure that systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model are brought to an acceptable level.³⁵⁵ Across this process,³⁵⁶ the measures adopted must be appropriate,³⁵⁷ that is, ‘suitable and necessary to achieve the intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and/or mitigation, whether through best practices, the state of the art, or other more innovative processes, measures, methodologies, methods, or techniques that go beyond the state of the art.’³⁵⁸ In this context, suitability, as one limb of the proportionality test,³⁵⁹ requires that a given measure be capable of effectively contributing to the objective of reducing systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to an acceptable level. The suitability of a measure may also be assessed by reference to how targeted it is, that is, the extent to which it relates to the identified risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .³⁶⁰ This is reflected in the GPAI Code of Practice’s preference for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to adopt targeted measures that address specific risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. without unduly impairing beneficial model capabilities.³⁶¹ 

103Necessity, in turn, requires that equal or superior safety or security outcomes cannot be achieved through alternative means that are less burdensome or more efficient.³⁶² In this respect, the notion of appropriateness under Article 55(1)(b) aligns with the reading of appropriate as used in Article 9(4),³⁶³ where it is understood as comprising two interrelated dimensions: first, effectiveness, in the sense that the measure must be capable of mitigating the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. in light of the current state of technology, and second, proportionality, in that the burden imposed by the measure must not be grossly disproportionate to the level of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. reduction achieved.³⁶⁴

104In addition to being informed by the principle of proportionality, the notion of appropriateness is also closely linked to the state-of-the-art condition.³⁶⁵ In selecting appropriate measures for assessing and mitigating systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the GPAI Code of Practice requires that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. consider ‘best practices, the state of the art, or other more innovative processes, measures, methodologies, methods, or techniques that go beyond the state of the art.’³⁶⁶ If a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. adopts only minimal measures while peers in the field typically implement more robust safeguards, such measures should be regarded as inappropriate. Improvements in developing mitigation measures corroborate that appropriateness is a dynamic standard,³⁶⁷ given that if more robust training techniques or improved testing practices become established in the industry, a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should incorporate them or have a compelling reason as to why not.³⁶⁸

2.1.2.3.2. Acceptable levels of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

105 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must adopt appropriate risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures until each identified systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and the overall systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is deemed to be acceptable.³⁶⁹ This formulation presupposes that some degree of residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. will remain following the implementation of mitigation measures. Residual risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. are those risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that are left over after mitigations have been implemented.³⁷⁰ What qualifies as an acceptable level of residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. is, in the first instance, determined by providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. themselves, in particular through the thresholds they establish to assess whether a given risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. is acceptable or requires further mitigation.³⁷¹

106Acceptability is defined elsewhere in terms of the two core dimensions of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , namely the severity and the probability of harm.³⁷² Accordingly, the higher the probability of harm and the more serious the nature and extent of the potential damage, the less likely it is that a risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. will be considered acceptable.³⁷³ Further insight into what constitutes acceptable or tolerable residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. can be derived from ISO Guide 51. According to this guidance, a number of factors may be taken into account when determining whether risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. is tolerable or acceptable, including prevailing societal values, the need to strike a balance between the ideal of absolute safety and what is achievable, the demands to be met by the product or system, and considerations such as suitability for purpose and cost-effectiveness.³⁷⁴  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may draw also on these factors in assessing whether residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. is acceptable in light of the objectives of the AI Act,³⁷⁵ which require balancing the promotion of innovation with a high level of protection of health, safety, and fundamental rights.³⁷⁶

107The inclusion of risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to fundamental rights within the notion of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , and thus into the process of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. acceptance determination,³⁷⁷ introduces a further layer of normative complexity.³⁷⁸ Specifically, it highlights the tension of trust in providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to independently assess how much encroachment on fundamental rights may be considered acceptable as residual risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .³⁷⁹ In this respect, the regulatory framework implicitly accepts that certain degrees of interference with fundamental rights may be tolerated.³⁸⁰ The AI Act does not provide a clear margin of appreciation for such harms, instead relying on general principles such as proportionality and the state of the art.

108Indeed, one consequence of linking acceptability to the state-of-the-art condition is that the level of mitigation expected for a given capability or risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. profile may increase over time as available safety and security practices evolve.³⁸¹ As more effective risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigation techniques become available, the continued presence of certain risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. may no longer be acceptable, meaning that the threshold of acceptable risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. evolves alongside other dynamic references in the AI Act, including measures that are appropriate and reflect the state of the art. In this respect, the AI Act does not require absolute safety, but rather a level of relative risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. reduction that reflects what is currently achievable.³⁸²

2.1.2.4. Timing of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures

109Article 55(1)(b) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that may arise across the model’s entire lifecycle, giving particular notice to those systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that arise during development, placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , and use of the model.

2.1.2.4.1. Development

110Although not defined by the AI Act, development can, in the context of Article 55, reasonably be understood to encompass the initial design and creation of the model, including data compilation, training, and fine-tuning.³⁸³  Systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may stem from choices made during this stage, even if their harmful effects only materialise after the model has been placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. . In that sense, risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. both arise from the development stage and may already be identifiable at that stage.

111While Article 2(8) excludes research, testing and development activities prior to market placement from the scope of the regulation,³⁸⁴ its impact is nuanced and does not preclude providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. from being required to conduct risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation before placing a model on the market.³⁸⁵ To ensure that, at the point of market placement, any residual systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. has been reduced to an acceptable level, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must undertake the necessary risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation processes during development.³⁸⁶ The exclusion of development activities as such does not remove the obligation to ensure compliance at the moment of placing the model on the market.³⁸⁷

2.1.2.4.2. Placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market.

112A GPAI model may be placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. in a variety of ways, including through ‘libraries, application programming interfaces (APIs), as direct download, or as physical copy.’³⁸⁸ In requiring providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to assess and mitigate risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that may stem from the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , attention should be paid both to systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that emerge after market placement and to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that arise because of the manner in which the GPAI model is placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. .³⁸⁹

113As to the former, the AI Act recognises that the full range and nature of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may become clear to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. only after market placement and in the course of the model’s interaction with users.³⁹⁰ To this end, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to conduct post-market monitoring in order to gather information about the model’s capabilities, propensities, affordances and effects, which may in turn inform whether the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. should be considered acceptable.³⁹¹ Post-market monitoring may reveal that the model’s capabilities, propensities or affordances have materially changed, such as through further post-training, access to additional tools, or increased inference compute.³⁹² It may also reveal developments that materially undermine the external validity of model evaluations previously conducted, materially improve the state of the art in evaluation methods, or otherwise suggest that the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment carried out was materially inaccurate.³⁹³

114 Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may also be influenced by the manner in which the GPAI model is placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. .³⁹⁴ Indeed, in assessing whether a GPAI model could present systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as part of the classification process under Article 51, the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ‘could take into account the way the model will be placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. or the number of users it may affect.’³⁹⁵ The Safety and Security Chapter of the Code of Practice also requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to implement safety mitigations that take account of the model’s release and distribution strategy, for example by ‘staging the access to the model, e.g. by limiting API access to vetted users, gradually expanding access based on post-market monitoring, and/or not making the model parameters publicly available for download initially.’³⁹⁶

2.1.2.4.3. Use of the model

115 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are also required to assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that may arise during the use of the model.³⁹⁷ A systematic reading of this obligation in light of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management process established for high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and the product-safety logic underpinning the AI Act’s risk-based approach³⁹⁸ indicates that Article 55(1)(b) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to also consider systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that may stem from the misuse of their models.³⁹⁹ Notably, the AI Act does not define what it means to use or misuse a (GPAI) model.⁴⁰⁰ It does, however, speak to what it means to use and misuse a high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .

116Article 9(2)(a) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. to assess risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that may arise when the AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. is used in accordance with its intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. . The concept of (intended) use can be traced back to the ISO Guide 51 on the inclusion of safety aspects in standards,⁴⁰¹ which defines intended use as ‘use in accordance with information provided with a product or system, or, in the absence of such information, by generally understood patterns of usage.’ ISO Guide 51 distinguishes between intended use, which is used synonymously with reasonably foreseeable use, and reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. .⁴⁰² While the AI Act does not define intended ‘use’ as such, it similarly frames the use of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. by reference to the intentions of the supplier, or, in the terminology of the AI Act, the intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. as envisaged by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. . Article 3(12) defines intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation.  as ‘the use for which an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. is intended by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , including the specific context and conditions of use’.⁴⁰³ The intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. of a system may be communicated by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. or deduced from the instructions for use Article 3(15) AI Act: ‘instructions for use’ means the information provided by the provider to inform the deployer of, in particular, an AI system’s intended purpose and proper use. , promotional or sales materials and statements, and the technical documentation prepared by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁴⁰⁴

117To meaningfully fulfil their obligations under Article 55(1)(b), providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must assess and mitigate risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. that may stem from the ways their GPAI model is used. However, given the very nature of such models as general-purpose and capable of being deployed across a potential vast range of contexts, GPAI models do not easily lend themselves to having an intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation.  as envisioned in Article 3(12).⁴⁰⁵ It is likewise unclear whether establishing such an intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. is a necessary precondition for the applicability and enforceability of Article 55(1). Even if a GPAI model cannot be said to have a clearly delineated intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. , it may nevertheless have an intended use, however broadly framed, as communicated by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. through system cards, model cards, and other public-facing documentation. Intended use may also be inferred from generally understood patterns of usage, which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are likely to monitor through post-market surveillance and market monitoring practices. The absence of a clearly defined intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. to which usage scenarios can be tethered, may indeed prompt providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to exercise greater effort in identifying, analysing and mitigating reasonably foreseeable systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . In this context, the requirement of reasonableness shifts the burden onto the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ‘research and understand the user environment and likely failure modes.’⁴⁰⁶

118 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. would also have to assess and mitigate reasonably foreseeable misuses, which includes use of a model in a way not intended by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. but which can result from reasonably foreseeable human behaviour.⁴⁰⁷ This means that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. have to consider ways their model can be intentionally or otherwise used in a manner not intended by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁴⁰⁸ While Article 55(1) does not explicitly refer to misuse, Recital 110 makes clear that ‘ systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. […] are influenced by conditions of misuse,’ noting that international approaches have identified the need to pay particular attention to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. arising from potential intentional misuse or from unintended issues of control relating to alignment with human intent. The Safety and Security Chapter of the Code of Practice requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to consider both intentional misuse and unintended model behaviour.⁴⁰⁹ 

119Intentional misuse of GPAI models involves the use of the model ‘by malicious actors to enable or scale harmful activities.’⁴¹⁰ The Code of Practice lists instances of intentional misuse, including, but not limited to, cyberattacks, development and use of CBRN capabilities, and large-scale disinformation.⁴¹¹ In identifying reasonably foreseeable risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. arising from intentional misuse, the Code of Practice requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to consider the potential number, capacity, and motivation of malicious actors to misuse a model as a potential source of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .⁴¹² This is further reflected in the requirements concerning model elicitation, under which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must undertake model evaluation that at least matches the elicitation capabilities of misuse actors relevant to the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. scenario.⁴¹³ 

120Unintended model behaviour refers to instances in which systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. arises from the use of the model in ways not anticipated by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , but not because the model is being deliberately used contrary to the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s intentions or instructions.⁴¹⁴ Rather, the model may behave ‘in ways that developers and users did not intend, or be unsafe in ways that could plausibly cause large-scale harm. This includes highly consequential accidents caused by inadequate capabilities, alignment, or safeguards.’⁴¹⁵ The Code of Practice recognises such instances of unintended model behaviour as falling within the broader category of loss of control.⁴¹⁶

2.1.3. Article 55(1)(c): Handling of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.

2.1.3.1. General remarks

121Article 55(1)(c) addresses the obligations of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. regarding the handling of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . According to this provision, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must ‘keep track of, document, and report without undue delay, to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and, as appropriate, to national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. , relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. and possible corrective measures to address them’.

2.1.3.1.1. Rationale

122The purpose of this provision is to ensure transparency regarding relevant incidents vis-à-vis the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. as the competent authority.⁴¹⁷ In addition, the provision aims to enable coordinated responses to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models, thereby preventing escalation after an incident, restoring (or preserving) the capacity to act and preventing further harm.⁴¹⁸ Furthermore, the provision facilitates the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ’s accumulation of knowledge, as incident reports provide up-to-date insights – particularly with respect to potential attack vectors⁴¹⁹ as well as potentially harmful patterns.⁴²⁰ Additionally, Article 55(1)(c) also promotes the exchange and dissemination of knowledge and thereby reduces potential information asymmetries between government and industry.⁴²¹ This also helps to inform future regulatory efforts as well as the development of best practices.⁴²² Finally, the provision ultimately helps to build public trust in AI technologies in general by enhancing proper oversight.⁴²³

2.1.3.1.2. Incident reporting obligations in EU law

123The obligation to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to authorities has been established across European legislation well before the AI Act. In particular, similar provisions can be found in many EU Regulations and Directives – for example in Article 87(1) Medical Devices Regulation (“MDR”)⁴²⁴, Article 23(1) NIS2 Directive (“NIS2”)⁴²⁵, Article 19(1) Digital Operational Resilience Act (“DORA”)⁴²⁶, and Article 14(1) Cyber Resilience Act (“CRA”)⁴²⁷. Article 33(1) GDPR likewise contains an at least comparable obligation.⁴²⁸

124Although these various provisions pursue different objectives in general,⁴²⁹ (and also use slightly different wording⁴³⁰) one might still be able to draw inspiration from them for the interpretation of Article 55(1)(c).⁴³¹ This is why the following discussion will, where relevant, refer to the reporting obligations under the aforementioned instruments.

2.1.3.1.3. Internal systematics of the AI Act and interaction with other EU legal instruments

125The obligation under Article 55(1)(c) entails interactions both within the AI Act and with other EU legal instruments. The most important interaction within the AI Act is that between Article 55(1)(c) and Article 3(49), as the latter defines the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ that is referred to in Article 55(1)(c). By its wording, however, the definition found in Article 3(49) refers only to AI systems, and not GPAI models (with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ). The Commission appears to assume the existence of a certain link between the two provisions.⁴³² The relationship between the two provisions will be examined in more detail below.

126Additionally, attention should be given to Article 73 of the AI Act on serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. reporting in the context of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . That provision contains a very similar obligation to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , though its content is considerably more detailed. The Commission has published its draft guidance on the reporting obligation under Article 73, to which this contribution will partly refer.⁴³³ Although that draft guidance expressly states that it is not intended to apply to Article 55(1)(c),⁴³⁴ it nevertheless contains some valuable indications on how the Commission interprets the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in Article 3(49), to which Article 55(1)(c) appears to refer.

127Notably, Article 73(9) governs some of the interaction with other EU legal instruments more precisely. The provision limits the reporting obligation for high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. by providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that are subject to Union legislative instruments that lay down their own reporting obligations equivalent to those in the AI Act. Under Article 73(1), those providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. only remain obliged to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. that directly or indirectly lead to an infringement of obligations under Union law intended to protect fundamental rights (Article 3(49)(c)).⁴³⁵ The reporting obligations under Article 19 DORA, Article 23 NIS2 and Article 14 CRA qualify as equivalent in that sense.⁴³⁶ High-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. subject to these obligations are therefore partially protected by Article 73(9) against duplicate reporting duties with regard to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. and thereby relieved of some administrative burden.⁴³⁷

128By contrast, no such rule exists under Chapter V in general or under Article 55 specifically for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Therefore, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may be subject to parallel reporting obligations under the CRA, the NIS2 Directive or the DORA under certain circumstances, resulting in an increased compliance burden. This means that, in any case, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must always report to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. – even if they already reported the same incident pursuant to another instrument to another institution.⁴³⁸

2.1.3.2. Relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.

2.1.3.2.1. Serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.

129As indicated earlier, Article 3(49) seems, at first glance, to define what constitutes a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. under the AI Act.⁴³⁹ According to this provision, ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ means ‘an incident or malfunctioning of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. that directly or indirectly leads to any of the following:

(a) the death of a person, or serious harm to a person’s health;

(b) a serious or irreversible disruption of the management or operation of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. ;

(c) the infringement of obligations under Union law intended to protect fundamental rights;

(d) serious harm to property or the environment.’

130As mentioned before, a difficulty arises from the fact that the definition expressly refers only to incidents or malfunctionings of AI systems, but not of (general-purpose) AI models. This opens three different pathways for interpretation. First, the definition could be understood to mean that an incident or a malfunctioning of a standalone GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can never give rise to a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. within the meaning of the AI Act – meaning that such an event could only occur if the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. has been integrated into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .⁴⁴⁰ Second, one could assume that this is simply a drafting error, with Article 3(49) to be read as if it stated: ‘an incident or malfunctioning of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. or GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.⁴⁴¹ Third, one might conclude that the definition in Article 3(49) is not exclusively determinative for Article 55(1)(c) but serves as the structural basis for a broader understanding of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.  aligned with Article 55’s overarching objective to assess and mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

2.1.3.2.1.1. Need for integration into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ?

131As explained above, the definition in Article 3(49) could be read to mean that standalone GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. cannot cause serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. within the meaning of the AI Act as long as they are not integrated into AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .⁴⁴² Recital 115, however, speaks against this interpretation, by referring to situations in which ‘the development or use of the model causes a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’. This recital therefore appears to assume that models themselves can also cause serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . Further, the GPAI Code of Practice supports an understanding under which GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. alone can cause serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , as it defines a ‘resolved serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ as a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. of a model…’.⁴⁴³ Moreover, both the Commission’s and the AI Board’s adequacy assessments of the GPAI Code of Practice also expressly refer to ‘ serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. of the model’.⁴⁴⁴ Finally, from a teleological perspective, interpreting the term serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. as exclusive to AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. could exclude some systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , such as loss of control over the model,⁴⁴⁵ from the reporting obligation, thereby undermining the rationale of Article 55 in assessing and mitigating systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

132On the other hand, it could be argued that a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. cannot occur without having people interact with the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and that this generally requires integration into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .⁴⁴⁶ This reading could be supported by the fact that other obligations in Article 55 explicitly establish a direct reference to the respective GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. (point (a): ‘perform model evaluation […] including conducting and documenting adversarial testing of the model’ (emphasis added); point (b): ‘assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level, including their sources, that may stem from the development, the placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , or the use of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market.  with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’ (emphasis added); point (d): ‘ensure an adequate level of cybersecurity protection for the general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’) (emphasis added), whereas this reference is missing in point (c) – indicating that the most direct point of reference in 55(1)(c) is not the model itself, but the respective AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. it is integrated in.

2.1.3.2.1.2. Application of Article 3(49) serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. definition?

133There are arguments to support a reading under which Article 3(49) is exclusively determinative for the concept of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. in Article 55(1)(c). One can argue that, in substance, Article 55(1)(c) seeks to ensure transparency in situations where particularly severe consequences have occurred and these consequences were caused by objects regulated by the AI Act. From a teleological point of view – having the aim of the AI Act to ensure a high level of health, safety and fundamental rights protection in mind – it should make no difference whether an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. or a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. caused the serious consequences.⁴⁴⁷ Moreover, a divergent definition of the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ within the AI Act could potentially lead to reporting gaps in cases where a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is integrated into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and where it is unclear whether the incident was (directly or indirectly) caused by the GPAI model or by the AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . That could be the case in instances where the model is integrated into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. that is not considered to be a high-risk system,⁴⁴⁸ and there is resultantly no obligation for system providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. .⁴⁴⁹ Additionally, in cases where the model is integrated into a high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. could argue that it was not the system but only the model that caused the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. .

134Diverging definitions of the notion of a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ could, in such cases, create additional uncertainty and result – in the worst case, especially if both concepts of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. were to be interpreted narrowly – in neither the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the potentially involved GPAI model nor the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the potentially involved AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. reporting the event. Moreover, pursuant to Article 73, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. do not need to directly report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. , but rather to the national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. .⁴⁵⁰ To avoid reporting gaps and to ensure the development of expertise and capability in the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ,⁴⁵¹ one could argue that a divergent interpretation of the term in Article 3(49) and in Article 55(1)(c) should therefore be avoided and that the definition found in Article 3(49) should also be exclusively determinative for Article 55(1)(c).

2.1.3.2.1.3. Article 3(49) as the structural basis for the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. definition in Article 55(1)(c)?

135It could also be argued that GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can cause serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , but that such incidents are not to be understood exclusively in the sense defined by Article 3(49).

136It seems likely that the Commission does not hold the definition provided in Article 3(49) to be exclusively determinative for Article 55(1)(c). The Commission ‘considers a “ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ” in the context of Chapter V AI Act as any incident or malfunctioning of a general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. that directly or indirectly leads to any of the events listed’ in Article 3(49)(a)–(d).⁴⁵² In this respect, the result of the Commission’s interpretation seems to be the same as if the definition in Article 3(49) were assumed to apply to GPAI models as well. At the same time, however, the Commission appears to adopt a broader understanding than determined in Article 3(49), as it sees ‘serious cybersecurity breaches’ as falling under the concept of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. of GPAI models under Article 55(1)(c) ‘due to their possible implications for the obligations provided for in Article 55(1), points (b) and (d)’.⁴⁵³

137The Code of Practice likewise suggests that the definition contained in Article 3(49) does not fully determine Article 55(1)(c)’s reporting obligation. Rather, the Code of Practice seems to indicate that what amounts to a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.  should be interpreted in light of the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that Article 55 seeks to capture. To this end, the Safety and Security Chapter of the GPAI Code of Practice, under Measure 9.3, like the Commission Guidelines,⁴⁵⁴ also refers to ‘a serious cybersecurity breach’ as a situation that triggers the reporting obligation under Article 55(1)(c) – a category that does not appear as a distinct one in Article 3(49).

138Additionally, the GPAI Code of Practice states that, for the purpose of assessing a model’s systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. under the Safety and Security Framework which signatories are required to set up, not only ‘ serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ but also ‘near misses’ can be relevant. For instance, near misses may serve as indicators that the Safety and Security Framework requires updating.⁴⁵⁵ On its face, one could argue that this only indicates that near misses need to be documented for the purposes of fulfilling Article 55(1)(a) and (b)’s obligations. According to the GPAI Code of Practice’s Glossary, however, a ‘near miss’ is defined as a situation in which ‘a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. could have, but ultimately did not, materialise’.⁴⁵⁶ This could be read as indicating that the materialisation of any harm is not necessary under the Code’s definition of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , given that the concept of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. forms part of the near miss definition itself. In relation to GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the Code therefore could be understood to interpret the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in a manner similar to the earlier Commission’s proposal, which, by using the formulation ‘might have led’, explicitly encompassed such near misses.⁴⁵⁷ One might therefore conclude that, while the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ continues to follow the definition in Article 3(49) for high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , it should be understood more broadly for GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to also include near misses. This reading is, however, open to challenge on two counts.

139An argument against this latter interpretation can be found in the GPAI Code of Practice itself, which clearly distinguishes between near misses and serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. in (all) other instances.⁴⁵⁸ Additionally, Measure 9.3 of the Safety and Security Chapter explicitly only sets out reporting timelines for situations in which the involvement of the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s model ‘(directly or indirectly) led’ – and not ‘might have led’ – to specific outcomes.⁴⁵⁹ Accordingly, as further discussed below,⁴⁶⁰ there likely exists no reporting obligation pursuant to Article 55(1)(c) for near misses under the Code of Practice and Article 55(1)(c).

140However, the legislative history of Article 3(49)⁴⁶¹ seems to imply that Article 3(49) should not necessarily serve as the structural basis for the concept of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. under Article 55(1)(c). The definition of a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in the initial Commission proposal did not contain the wording that the incident must be that ‘of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’.⁴⁶² This addition was only introduced in the Council’s proposal⁴⁶³ and ultimately found its way into the final legal text of the AI Act. Because the first rules on GPAI (at that time called ‘ general-purpose AI systems Article 3(66) AI Act: ‘general-purpose AI system’ means an AI system which is based on a general-purpose AI model and which has the capability to serve a variety of purposes, both for direct use as well as for integration in other AI systems. ’) were also incorporated into the AI Act for the first time during the Council’s proposal stage,⁴⁶⁴ the simultaneous addition of ‘of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’ instead of ‘of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. or GPAI model’ into the definition might seem deliberate.⁴⁶⁵ However, it seems equally plausible that the absence of the wording ‘or a GPAI model’ may have been a drafting oversight following the introduction of the first special rules on general-purpose AI systems Article 3(66) AI Act: ‘general-purpose AI system’ means an AI system which is based on a general-purpose AI model and which has the capability to serve a variety of purposes, both for direct use as well as for integration in other AI systems. . It can additionally be argued that, at the time of adding provisions on GPAI, the wording ‘of an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’ was also intended to cover GPAI, which at that stage in the legislative process – as stated above – was referred to as ‘general purpose AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ’.⁴⁶⁶

2.1.3.2.1.4. Synthesis

141Some ambiguity clearly persists. The most convincing arguments, however, appear to support a partial reliance on the definition of a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in Article 3(49) as a structural basis, while recognising that Article 55(1)(c) ultimately addresses different risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. than the ones found in high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. contexts (Article 73 AI Act). For the practical application of Article 55(1)(c), and based on a purposive reading of it, the following approach seems appropriate: as a general rule, the definition set out in Article 3(49), interpreted so as to include GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , should apply.⁴⁶⁷ In view of the AI Act’s objective to ‘ensure a high level of protection of health, safety and fundamental rights’, it makes sense to establish a reporting obligation concerning the consequences listed in Article 3(49), irrespective of whether they result from a high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. or a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ⁴⁶⁸ (or both) and to keep the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. informed in this regard. This approach prevents uncertainty and reporting gaps in cases where a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is integrated into a non-high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and encompasses the cases in which a model was not integrated into a system.

142At the same time, however, there are good reasons to argue that the definition of a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ should be extended in the context of Article 55(1)(c), given the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. inherent to GPAI models presenting systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . This is also indicated by both the Commission GPAI Guidelines and the GPAI Code of Practice.⁴⁶⁹ The most obvious example being the ‘serious cybersecurity breach’, which both the Code and the Guidelines refer to.⁴⁷⁰ 

143Beyond that, but probably less convincing,⁴⁷¹ one could argue that an even broader interpretation appears justified, encompassing additional consequences insofar as they materially increase the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model – as serious cybersecurity breaches arguably do – or they constitute a materialisation of a systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . More generally, insofar as the obligation to keep track of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. serves to inform the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment,⁴⁷² there also seem to be reasons to require the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to keep track of, document and report all incidents that materially increase or lead to the materialisation of systematic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. – and not just serious cybersecurity breaches, as the Commission GPAI Guidelines and the GPAI Code of Practice propose.⁴⁷³ Expanding Article 55(1)(c) in this way arguably seems important because incidents that materially increase systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are not captured by Article 3(49)’s definition – which is tailored to downstream harm caused by AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . Ultimately, the reporting obligation serves not only the purpose of making the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. aware of previously unidentified risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. – expanding the obligation to report to include material increases of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model ensures that both also become aware of the (near-)materialisation of already known risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. which, in light of the incident, may no longer be seen as adequately mitigated.

144There are also policy considerations that might support this broader interpretation. California Transparency in Frontier Artificial Intelligence Act (“SB-53”) – which, as the first comprehensive frontier AI safety statute enacted in the United States undoubtedly carries international significance – similarly addresses the inherent risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of highly capable models within its reporting provisions.⁴⁷⁴ It is particularly noteworthy in this regard that the scope of the reporting obligation in SB-53 is confined to large frontier developers and models⁴⁷⁵ – and thus differs structurally from the general incident reporting framework of the AI Act, which was conceived primarily with high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. in mind. The definition of a ‘critical safety incident’ under SB-53 encompasses, for instance, ‘[h]arm resulting from the materialization of a catastrophic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’ (emphasis added) and cases in which a ‘frontier model that uses deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ’ (emphasis added).⁴⁷⁶ It is precisely this model-specific tailoring of SB-53’s incident reporting concept that makes the comparison structurally apt.

145One might be tempted to introduce these policy considerations – at least indirectly – into the legal interpretation of the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in Article 55(1)(c) via Article 56(1), which mandates the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to ‘take into account international approaches’. However, this line of argument faces several objections. First, the obligation to take international approaches into account binds the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. only in the specific context of encouraging and facilitating the drawing up of codes of practice – not in the interpretation of the AI Act itself.⁴⁷⁷ Second, even if one were to argue that the GPAI Code of Practice – having been recognised by the Commission as adequate – informs the interpretation of Article 55(1)(c), this reasoning must ultimately be abandoned in light of the chronological sequence: SB-53 was enacted after the GPAI Code of Practice had already been finalised. In the end, this comparative argument is therefore one that carries no normative force – though it might retain persuasive value from a policy perspective.

146There is also a teleological argument to extend the concept of a ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ within the meaning of Article 55(1)(c) to cover near misses as well.⁴⁷⁸ This expansion is suggested by the GPAI Code of Practice’s definition of a near miss and would also be supported by a purposive reading of Article 55(1)(c). In view of the expected information asymmetry between providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. , and the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ’s need to be able to detect harmful patterns and emerging risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. vectors at an early stage, it may be inappropriate in such a sensitive area as systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation to wait until risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. have materialised.⁴⁷⁹

147However, several arguments weigh against including near misses into the definition of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . First, earlier drafts of the definition of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. included wording that would have covered near misses (‘might have led’). This wording was removed during the drafting process, signalling the EU legislature’s intent to keep the two concepts separate.⁴⁸⁰ Likewise, the GPAI Code of Practice refers only to reporting obligations in cases where the involvement of the model led to a particular outcome.⁴⁸¹ It also clearly distinguishes – as noted above, notwithstanding the broader definition in the Glossary – between near misses and serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , suggesting that near misses fall outside the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. definition.⁴⁸² The argument that (all)⁴⁸³ near misses need to be reportable to inform the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. at an early stage is also not particularly persuasive. That is because providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may need to document and assess near misses under Article 55(1)(a) and (b),⁴⁸⁴ enabling the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to request that information pursuant to Article 91. Further, near misses trigger an update of a signatory’s Safety and Security Framework⁴⁸⁵ in cases where they are ‘likely to indicate that the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from at least one of [the signatories] models are not acceptable have occurred.’⁴⁸⁶ Such updates must be notified to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. within five business days.⁴⁸⁷ As a result, the view that near misses are not captured by Article 55(1)(c)’s reporting obligation is more compelling than the alternative.

148Since, as demonstrated above, the interpretation of the term ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in Article 55(1)(c) is built on the definition set out in Article 3(49) as its structural basis, the following analysis is structured in accordance with that definition. Particular attention will be drawn at the relevant points to the specific considerations that arise for GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. in this context.

2.1.3.2.1.5. Incident or malfunctioning

149Article 3(49) defines a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. as ‘an incident or malfunctioning’. The Commission likewise refers to this phrase in its GPAI Guidelines.⁴⁸⁸ However, what is to be understood by ‘incident or malfunctioning’ is not further defined in the AI Act.⁴⁸⁹ There seem to be two possible pathways for interpretation.

150On the one hand, because the reporting of an incident does not amount to an admission of wrongdoing⁴⁹⁰ and because the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. needs to be kept up to date regarding existent and emerging systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. within the Union, one could – from a teleological point of view – argue that what matters is merely whether the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. was somehow involved.⁴⁹¹ The term ‘incident’ would then not carry an additional independent meaning as long as the causality requirement is fulfilled. This understanding also seems to be indicated in the GPAI Code of Practice, which refers more generally to the ‘involvement’ of the model (directly or indirectly) leading to a specified outcome, rather than an incident or malfunctioning of the model.⁴⁹²

151On the other hand, it also seems reasonable to interpret the term ‘incident’ as entailing an additional limiting criterion – such that not every causal connection between a GPAI model and a specified outcome would be sufficient to trigger the reporting obligation.⁴⁹³ Likewise, the Commission, in its draft guidance on the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. reporting obligation for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , states that an incident ‘is a not planned/programmed deviation in the characteristics of performance’.⁴⁹⁴ Under this – more narrow – interpretation, to better understand the notion of an ‘incident’ it is useful to look at several other EU instruments that contain a definition of that term.

152A definition of the term ‘incident’ can be found in several other EU instruments that contain similar reporting obligations.⁴⁹⁵ Looking at those instruments – notably the MDR, the NIS2 Directive and the DORA – one might be able to carefully draw some interpretative inspiration for the AI Act.⁴⁹⁶ First, translated to the governance of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the MDR indicates that an incident could be said to occur only where the characteristics or performance of the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. deteriorate.⁴⁹⁷ The fact that use-errors are explicitly mentioned in the MDR but not in the AI Act seems to be a knife that cuts both ways. One could argue that the fact that it is expressly mentioned in the MDR speaks against adopting this approach in the AI Act. On the other hand, one could argue that this is a general principle within reporting obligations throughout the EU legal order and it was just expressly mentioned for clarification in the MDR. Second, the NIS2 Directive’s definition⁴⁹⁸ likewise refers to a deterioration of certain characteristics of a service (or data). Third, the DORA⁴⁹⁹ implies that only unplanned events are captured and that also a series of linked events can suffice for the definition to be applicable.⁵⁰⁰ Similarly, the draft guidance on the reporting obligation under Article 73 lists some examples of incidents and malfunctions that largely feature unplanned events, namely misclassifications, significant drops in accuracy, temporary system downtime and unexpected system behaviour.⁵⁰¹

153At the same time, however, applying ‘unplanned’ as a delimiter to GPAI models appears more difficult because their behaviour is hard to predict and they exhibit no genuine intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. ,⁵⁰² so there is, in a sense, no baseline against which deterioration could be measured. What also remains less clear under this approach is why – compared to the Commission’s proposal of the AI Act⁵⁰³ – the term ‘malfunctioning’ was added as an alternative element of the definition.⁵⁰⁴ In the MDR, for example, a malfunction is merely a subcategory of an incident.⁵⁰⁵ The added term in the AI Act likely primarily serves a clarifying function, for example, intended to make clear that not only events triggered by malicious actors (as the term ‘incident’ could be understood to imply) are meant to be captured.⁵⁰⁶

2.1.3.2.1.6. Causal connection

154For the reporting obligation under Article 55(1)(c) to be triggered, a causal connection must exist between the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. concerned and one of the qualifying outcomes. The precise manner in which that causal connection is to be established, and between which specific elements it must exist, remains unclear.

155If one follows the approach that the concept of ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ in Article 3(49) is to be treated as the structural basis for Article 55(1)(c), the incident or malfunction would need to be in a causal connection (‘directly or indirectly leads to’) to at least one of the outcomes specified in Article 3(49) (or to another situation in which a systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. has materially increased or materialised). The Commission also partly refers to this same wording in its GPAI Guidelines – stating that apart from aforementioned cybersecurity breaches, it considers ‘a “ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ” in the context of Chapter V AI Act as any incident or malfunctioning of a general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. that directly or indirectly leads to any of the events listed in the corresponding definition for AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. in Article 3(49), points (a) to (d), AI Act.’⁵⁰⁷ 

156The GPAI Code of Practice requires signatories to report ‘if the involvement of their model (directly or indirectly) led to’ (emphasis added) the further specified outcomes.⁵⁰⁸ Therefore, the Code of Practice seems to indicate that the causal connection need not exist between an incident or malfunctioning of the model and a specified outcome, but rather it suffices if the model is in some way causally involved. The outcome of this interpretation would be the same as under the previously discussed interpretation, if one does not understand the ‘incident or malfunction’ component of the definition to be a limiting criterion.⁵⁰⁹

157Apart from that ambiguity with regard to the causal link between the model and a specified outcome, a different understanding and concept of the causality requirement seems to be implied both in the GPAI Code of Practice and in the Commission’s GPAI Guidelines with regard to serious cybersecurity breaches. Measure 9.3(2) of the Safety and Security Chapter of the GPAI Code of Practice requires for the reporting obligation to be triggered that the involvement of the signatory’s model ‘led to … a serious cybersecurity breach, including the (self-)exfiltration of model weights and cyberattacks’.⁵¹⁰ This formulation largely resembles the Commission’s GPAI Guidelines, which provide that ‘cybersecurity breaches related to the model or its physical infrastructure, including the (self-)exfiltration of model parameters and cyberattacks’ are to be reported.⁵¹¹ The latter formulation clarifies that it suffices that the model is somehow involved (‘related to the model or its physical infrastructure’, emphasis added). This implies that certain outcomes – especially those increasing the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model – do not necessarily need to be caused by the model but can also be happening to the model.

158Regardless of whether one requires a link between the model’s involvement or an incident or malfunction of the model and a qualifying outcome, and following the above, the interpretation of the causation concept embedded in the ‘directly or indirectly leads to’ formulation seems to become particularly relevant with regard to the outcomes enumerated in Article 3(49) as well as materialisations of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . This causal relationship will in many cases be an indirect one, as a model will rarely be capable of bringing about the outcomes referred to in Article 3(49) without prior integration into an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . In its draft guidance on the reporting obligation under Article 73, the Commission defines an incident or malfunction as directly or indirectly causal ‘if, without it, the harm in its concrete form would not have occurred (or reasonably likely respectively more probable not to have occurred).’⁵¹² The draft guidance further clarifies that secondary effects suffice, possibly even multiple steps later in the chain – an example being an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. that provides incorrect medical imaging, leading to an incorrect diagnosis, ultimately leading to harm to a patient.⁵¹³ Finally, the Commission intends to limit causations to cases of system use corresponding to its intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. ⁵¹⁴ or reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. .⁵¹⁵ 

159This definition set out in the draft guidance on Article 73 comprises two components – a factual one and a normative one – of which, however, only one is directly transposable in the GPAI model context. What may be directly transposable is the factual component,⁵¹⁶ understood in the sense of the conditio sine qua non formula familiar to most EU legal systems.⁵¹⁷ The relevant question to be asked is therefore whether the harm in its concrete form – so, especially the outcomes mentioned in Article 3(49) – would not have occurred (or reasonably likely respectively more probable not to have occurred) had a different model been used instead or had the model not experienced an incident or a malfunction.⁵¹⁸ 

160The second component, however, appears to be a normative one.⁵¹⁹ The limitation in the draft guidance on Article 73 to cases in which the AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. is used in accordance with its  intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation.  or reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems.  seems to rest on the normative premise that the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. management obligations imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 9 are themselves confined to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. arising from use in accordance with the system’s intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. and under conditions of reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. .⁵²⁰  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might therefore be subject to reporting obligations under Article 73(1) only in relation to materialisations of the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. they are required to assess and mitigate under Article 9. That logic cannot be directly transposed to GPAI models. Given their significant generality and ability to form the basis for a wide range of downstream systems and applications,⁵²¹ the concepts of intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. and reasonably foreseeable misuse Article 3(13) AI Act: ‘reasonably foreseeable misuse’ means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems. might not map neatly onto GPAI models.⁵²² The risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. for which a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. bears responsibility are, however, defined by Article 55(1)(a), (b) and (d), which perform an at least similar structural function as Article 9(2)(a) and (b) in the high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. framework.⁵²³ Consistent with that rationale, the reporting obligation under Article 55(1)(c) should not extend to every serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. that is factually connected to the development or use of a GPAI model. Rather, it should be limited to incidents that materialise within the possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. was required to assess and mitigate under Articles 55(1)(a), (b) and (d).⁵²⁴ This conclusion is supported by the wording of Recital 115, which conditions the reporting obligation on situations in which, ‘despite efforts to identify and prevent risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. related to a general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. that may present systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , the development or use of the model causes a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ (emphasis added). Additionally, one might argue that a purposive reading suggests that incidents that do not fulfil this normative criterion do not fall within the purpose and objective of Article 55(1)(c) to inform the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. identification, assessment and mitigation pursuant to Article 55(1)(a) and (b).

2.1.3.2.2. Specified outcome

161According to the definition in Article 3(49), the obligation to report a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. applies if one of the specified outcomes actually occurs.

2.1.3.2.2.1. Death or serious harm

162The first outcome covered is the death of a person or serious harm to a person’s health. In most cases, a person’s death (as well as the further defined harms in points (b) to (d)) will likely be directly caused by an AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. rather than a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. – for example, where the system’s output triggers a mechanical reaction.⁵²⁵ However, cases cannot be ruled out in which it can be demonstrated that it was not the AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. but rather (or additionally) the underlying GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that was at least indirectly responsible for a person’s death.⁵²⁶

163The AI Act does not expressly define what exactly constitutes serious harm in the sense of Article 3(49). To begin with, the open wording suggests that not only physical harm but also mental health harm is likely to be covered.⁵²⁷ Some authors suggest that the parameters for determining whether harm qualifies as serious could include the intensity, extent and severity or duration of the harm as well as the consequences for the person’s daily life.⁵²⁸ To ensure at least some normative anchor, it also seems possible to draw guidance from Article 2(58) MDR, which provides a more detailed definition of the term ‘serious adverse event’ and bears certain resemblances with the AI Acts definition in Article 3(49)(a)⁵²⁹. According to that provision, a serious adverse event is any adverse event that led to the death or a ‘serious deterioration in the health of the subject Article 3(58) AI Act: ‘subject’, for the purpose of real-world testing, means a natural person who participates in testing in real-world conditions. ’ resulting in further defined outcomes. Although this definition concerns the consequences of a serious health deterioration, it nevertheless seems plausible to qualify at least the listed outcomes as serious harm to a person’s health within the meaning of the AI Act. These outcomes are: ‘(i) life-threatening illness or injury, (ii) permanent impairment of a body structure or body function, (iii) hospitalisation or prolongation of patient hospitalisation, (iv) medical or surgical intervention to prevent life-threatening illness or injury or permanent impairment to a body structure or a body function (v) chronic distress’.⁵³⁰

2.1.3.2.2.2. Disruption of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557.

164Article 3(49) also defines ‘ serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ to include ‘serious and irreversible disruption of the management or operation of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. ’.

165Article 3(62) defines the term critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557.  with reference to Article 2(4) of the Critical Entities Resilience Directive (“CERD”)⁵³¹ which defines ‘ critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. ’ as ‘an asset, a facility, equipment, a network or a system, or a part of an asset, a facility, equipment, a network or a system, which is necessary for the provision of an essential service’, with an essential service defined as ‘a service which is crucial for the maintenance of vital societal functions, economic activities, public health and safety, or the environment’ (Article 2(5)).

166The notion of ‘serious and irreversible disruption’ in Article 3(49) remains unclear, however. Particular attention must be paid to the fact that both elements must be fulfilled cumulatively (‘and’). The question of when a disruption is serious is answered inconsistently in legal literature. Some authors suggest that the decisive factor is whether the effects of the disruption are or could be severe.⁵³² Others propose looking at factors such as intensity, magnitude and extent, duration and reversibility, impairment of the supply situation (types of goods affected, number of persons affected), threats to public safety and order, and societal impacts.⁵³³ Although both approaches seem plausible, they both lack a clear normative anchor. Some authors therefore recommend drawing on Article 15(1) CERD, which provides a non-exhaustive list of parameters that might be used to assess whether a disruption of the provision of an essential service was significant.⁵³⁴ This approach appears most persuasive given the AI Act’s explicit reference to the CERD.⁵³⁵ Article 15(1) CERD lists: ‘(a) the number and proportion of users affected by the disruption; (b) the duration of the disruption; (c) the geographical area affected by the disruption, taking into account whether the area is geographically isolated’.

167Additionally, in its draft guidance on the reporting obligation pursuant to Article 73, the Commission gives some examples of disruptions that should be considered serious. Those are (i) ‘The disruption might result in an imminent threat to life or the physical safety of a person, including through serious harm to the provision of basic supplies to the population or the exercise of the core function of the State’, as well as (ii) ‘Destruction of key infrastructure’ and (iii) ‘Disruption in social or economic activities’.

168It is important to note that the parameters listed in Article 15(1) CERD address only the significance of the disruption in the provision of an essential service.⁵³⁶ Unlike the CERD, however, the definition in Article 3(49) also covers disruptions in the management of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. .⁵³⁷ Where such disruptions likewise affect the operation of the infrastructure, the parameters mentioned above might nonetheless be applied. What remains unclear, however, is how to assess the serious disruption of the management of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. if it does not also impact the operation. One possible approach would be to consider management disruptions as serious if – absent intervention by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. – they could subsequently lead to a serious disruption in the operation of the critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. .⁵³⁸

169Moreover, the disruption must not only be serious, but also irreversible.⁵³⁹ The concept of irreversibility is not reflected in current EU legislation on critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. , and it is also unclear how it differs from destruction.⁵⁴⁰ Furthermore, it remains unclear why providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are, pursuant to Article 55(1)(c), required to keep track of, document and report possible corrective measures with respect to disruptions that are, by definition, irreversible anyway. One possible approach would be that the assessment of the seriousness already encompasses considerations of reversibility. This, however, is countered by the wording,⁵⁴¹ which clearly presupposes the existence of disruptions that are serious but not irreversible. To give the term an independent meaning, it might be understood as to denote a disruption that cannot be remedied through ordinary maintenance work for which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are expected to provide at all times.⁵⁴² Another line of reasoning would hold that any disruption, once it has occurred, is never fully reversible and that the focus should therefore be on the consequences. An illustrative example would be a large-scale power grid failure leading to a prolonged blackout across an entire Member State. Such an event would probably qualify as a serious disruption. It would, however, be considered irreversible under the latter interpretation only if, for example, as a result of the blackout, persons in a nearby hospital were to die. In other words: a disruption might be regarded as irreversible when, despite its eventual rectification, further negative effects remain.

170The latter view also appears to be shared by the Commission in its draft guidance on Article 73, which states that, to qualify as a serious disruption, the following aspects should be taken into account: (i) ‘The disruption requires rebuilding of physical infrastructure or destroys specialized equipment which is not readily available’ as well as (ii) ‘Contamination of water, soil or air’ and (iii) ‘Loss or corruption of essential records – such as patient data, civil registries, or financial transactions – that cannot be reliably restored or reconstructed’. Also mentioned are (iv) ‘Permanent disablement of a critical node, such as a rail junction, power substation, or landing station, that cannot be repaired or replaced without years-long lead times’ and (v) ‘Loss of a space-based asset (e.g. Global Navigation Satellite System or communications satellite) whose destruction vacates its orbital slot or frequency and cannot be replaced without an extended replacement procedure that typically lasts years.’⁵⁴³

2.1.3.2.2.3. Infringement of EU law to protect fundamental rights

171Furthermore, covered by Article 3(49) is ‘the infringement of obligations under Union law intended to protect fundamental rights’. In the European Parliament’s earlier proposal,⁵⁴⁴ a narrower version of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment.  definition covered only ‘a breach of fundamental rights protected under Union law’.⁵⁴⁵ The enacted definition is, by its wording, considerably wider and risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. becoming amorphous, as it captures not only infringements of fundamental rights themselves but of all ‘obligations’ under Union law intended to protect fundamental rights.⁵⁴⁶

172This might lead to the challenge⁵⁴⁷ that obligations under, in particular, the GDPR qualify, with the result that every violation of GDPR obligations, for example every personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. breach within the meaning of Article 4(12) GDPR, could, in theory, constitute a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. .⁵⁴⁸ This would produce the contradictory outcome that a personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. breach would always trigger the reporting obligation under Article 55(1)(c), while it might, by contrast, not even require notification under the GDPR itself because of the exception in Article 33(1) GDPR (i.e. ‘unless the personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. breach is unlikely to result in a risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to the rights and freedoms of natural persons’).⁵⁴⁹

173More generally, it makes little sense to capture only serious health injuries or, for example, only serious and irreversible disruptions of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. on the one hand, while at the same time, on the other hand, allowing any violation of ordinary statutory provisions to suffice elsewhere – provided that they are ‘intended to protect fundamental rights’ and relate to (another) fundamental right. To avoid such contradictory outcomes, either a certain degree of severity of the infringement should be required⁵⁵⁰ or the term ‘intended to protect’ should be interpreted narrowly.⁵⁵¹ In assessing the severity of the infringement, it would be reasonable to take guidance from the other categories in Article 3(49), not least because it could create an inconsistency if certain obligations were recognised as aiming to protect ‘health’ or ‘property’ as fundamental rights, and breaches of those obligations would therefore fall within Article 3(49)(c) yet would fail to meet the higher thresholds required in Article 3(49)(a) (‘death’ and ‘serious harm’) and (d) (‘serious harm’). After all, an overly broad interpretation would also run counter to the very purpose of the reporting obligation, as it would force the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to expend resources unnecessarily and could impair its ability to respond appropriately to genuinely serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . Such an information overload should be avoided through a restrictive interpretation.⁵⁵²

174In its draft guidance for Article 73, the Commission gives some examples for infringements covered by the definition. These include (i) ‘An AI based recruitment system excludes candidates based on ethnicity or gender’ as well as (ii) ‘A credit scoring system excludes certain categories of persons, such as those having a name from a certain region or living in certain neighbourhoods’ and (iii) ‘A biometric identification Article 3(35) AI Act: ‘biometric identification’ means the automated recognition of physical, physiological, behavioural, or psychological human features for the purpose of establishing the identity of a natural person by comparing biometric data of that individual to biometric data of individuals stored in a database. system frequently wrongly identifies people of different ethical background.’⁵⁵³

2.1.3.2.2.4. Serious harm to property or environment

175Lastly, Article 3(49) covers ‘serious harm to property or the environment’.

176The AI Act provides no further guidance on what is to be understood under this term. Some authors propose considering ‘the amount of damage’ for property damage and ‘the financial cost of removal, but also, for example, […] the danger to humans, animals, plants and their habitat’ for environmental damages.⁵⁵⁴ According to others, relevant factors include, for example, the value of the affected goods, the irreversibility of the damage, the number of persons affected (including future generations for environmental damage), and the extent of the impact on natural ecosystems.⁵⁵⁵ Other authors argue that one may look, inter alia, at the intensity, magnitude, and extent of the damage; the goods affected, the duration and reversibility of the damage, the number of persons affected as well as the broader societal consequences.⁵⁵⁶ To assess the seriousness of environmental harm, one could also draw on the ISO 14001 standard for environmental management systems, which contains categories for both ‘minor environmental impact’ and ‘major environmental impact’.⁵⁵⁷

177The Commission’s draft guidance on Article 73 further sets out the following parameters for assessing the seriousness of harm to property:⁵⁵⁸ (i) ‘The economic impact, including cost of repair or replacement. The damage to property is deemed to be serious if the damage or destruction impairs the intended usability or substance of the property to such an extent that it can no longer be used for its intended purpose Article 3(12) AI Act: ‘intended purpose’ means the use for which an AI system is intended by the provider, including the specific context and conditions of use, as specified in the information supplied by the provider in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation. . The amount of damage, the cost of repair or the reduction in value are not decisive in this respect, but should in any case exceed 5% of the purchase price’; (ii) ‘The cultural, or historical significance of the property’; (iii) ‘The extent to which the property loss or damage affects the livelihood or quality of life of individuals or communities.’; (iv) ‘The permanence of the damage, including whether the property can be restored to its former state.’ as well as (v) ‘The ripple effects of the damage, such as its impact on surrounding areas or related operations.’⁵⁵⁹

178With respect to the seriousness of harm to the environment, the draft guidance lists the following parameters: (i) ‘the baseline condition of the affected environment’; (ii) ‘whether the damage is long-lasting, medium-term or short-term’; (iii) ‘the extent of the damage’ as well as (iv) ‘the reversibility of the damage.’ Examples mentioned are the ‘[c]ontamination of environmental resources’ and ‘[d]isruption of natural ecosystems’.⁵⁶⁰

2.1.3.2.2.5. Increase or materialisation of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain.

179As mentioned above, there are arguments to assume that the concept of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. needs to be understood more broadly with regard to Article 55(1)(c), compared to Article 3(49).⁵⁶¹ This line of argumentation is built upon the statement in the Commission’s GPAI Guidelines that the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. considers Article 55(1)(c) to cover ‘serious cybersecurity breaches related to the model or its physical infrastructure, including the (self-)exfiltration of model parameters and cyberattacks due to their possible implications for the obligations provided for in Article 55(1), points (b) and (d)’. Additionally, as the GPAI Code of Practice emphasises,⁵⁶² the main purpose of the obligations laid down in Article 55 is to assess and mitigate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Since not only serious cybersecurity breaches but also other increases and materialisations of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model have implications for the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s obligations at least under Article 55(1)(b),⁵⁶³ it makes sense to have the obligation under Article 55(c) encompass not only the outcomes mentioned in Article 3(49) but also material increases and materialisations of the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model identified by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. pursuant to Commitment 2 of the Safety and Security Chapter of the GPAI Code of Practice.

180Leaving aside the question of whether the inclusion of ‘serious cybersecurity breaches’ in the Commission’s GPAI Guidelines and the GPAI Code of Practice support a broader reading of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. concept in Article 55(1)(c), it remains to be determined which events precisely qualify as serious cybersecurity breaches. The GPAI Guidelines and the GPAI Code of Practice offer some clarification insofar as both indicate that the term is to be understood as ‘including the (self-)exfiltration of model weights and cyberattacks’.⁵⁶⁴ Beyond these examples, however, the contours of the concept remain rather unclear. Pending further guidance, the assessment should be informed by whether the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model has been materially increased by the breach.⁵⁶⁵ This will arguably be the case primarily where model weights have been exfiltrated. By contrast, not every unsophisticated cyberattack targeting the model should trigger the reporting obligation, so as to avoid an information overload at the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. . A further question concerns the interplay between the obligation to report serious cybersecurity breaches and the exemption in Commitment 6 of the Safety and Security Chapter of the GPAI Code of Practice. This states that a model is exempt from Commitment 6 where a more capable model’s parameters are available for download. Following this logic – an exemption from security measures where a more capable model is freely available – one might argue that a cybersecurity breach cannot be considered ‘serious’ where the model concerned already falls within the exemption under Commitment 6. Overall, further regulatory guidance on the precise definition of the term appears desirable.

181If one were to follow the broader reading of Article 55(1)(c), which includes not only serious cybersecurity breaches as reportable incidents, but also other instances of material increases in systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model,⁵⁶⁶ it remains open as to when such an increase triggers the reporting obligation. A parallel exists in the obligation of signatories to update their Safety and Security Model Reports pursuant to Measure 7.6 of the Safety and Security Chapter. According to that measure, signatories must update their Safety and Security Model Report ‘if they have reasonable grounds to believe that the justification for why the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model are acceptable … has been materially undermined’.⁵⁶⁷ Under this broader reading, both signatories and non-signatories should be able to orientate themselves by reference to this criterion.⁵⁶⁸ 

182It must be acknowledged, however, that this reading sits in some tension with the earlier conclusion that near misses likely fall outside Article 55(1)(c)’s scope.⁵⁶⁹ A near miss may, in a given case, be precisely the kind of event that evidences a material increase in systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . One could argue, however, that this overlap is narrower than it first appears. That is because not every near miss will indicate that the model’s systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. has materially increased: A near miss in which the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigations worked as intended – for example a successfully repelled cyberattack – may confirm the risk-acceptability justification rather than indicating that is has been undermined.⁵⁷⁰ The aforementioned tension can therefore be mostly resolved if only those near misses that reveal risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. mitigations to be weaker than assumed or surface capabilities that were not previously accounted for qualify as material increases in the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model.

183Additionally, and against this approach, however, one might ask what difference would then remain between the obligation to update and provide the Safety and Security Model Report to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the reporting of material increases in systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. under Article 55(1)(c). An important difference lies in the fact that the update of a Safety and Security Model Report must be completed within a ‘reasonable amount of time’ after a signatory has identified the necessity of an update, followed by a further period of five business days within which the updated report must be sent to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. .⁵⁷¹ Given the extensive information required in such a report – see Measures 7.1 to 7.5 of the Code of Practice – it will accordingly take some time before the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. receives the relevant information. Treating material increases in systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model as serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. under Article 55(1)(c) would therefore have the advantage that information could reach the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. earlier, without requiring a full Safety and Security Model Report update, thereby enabling the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to react more promptly where necessary. Depending on how the terms ‘incident’ and ‘malfunctioning’ are interpreted, it must further be noted that the obligation to provide an updated Safety and Security Model Report also encompasses deliberate changes made to the model by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ,⁵⁷² whereas the obligation under Article 55(1)(c) may not extend to such changes.⁵⁷³

2.1.3.2.3. Relevant information

184It is important to note that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are required to report, keep track of, and document not the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. as such, but ‘relevant information’ about it. The Commission has published a reporting template for serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. involving GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. in November 2025, which is intended to ‘serve as a means to demonstrate compliance with Article 55(1), point (c), of the AI Act as part of Commitment 9 of the Safety and Security Chapter of the General-Purpose AI Code of Practice’.⁵⁷⁴ 

185The reporting template and the GPAI Code of Practice provide a helpful overview of the types of information that can be considered relevant in this context. Reports must cover: (i) the dates of the incident or best approximations thereof; (ii) the resulting harm and those affected; (iii) the chain of events leading to the incident; (iv) the model involved; (v) available material documenting the model’s involvement; (vi) the signatory’s response or intended response; (vii) the signatory’s recommendations to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and, where applicable, national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. ; (viii) a root cause analysis, including the model’s relevant outputs, contributing factors, and any failures or circumventions of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigations; and (ix) any patterns from post-market monitoring reasonably connected to the incident, including data on near misses.⁵⁷⁵

186As noted above, near misses themselves arguably do not qualify as a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. in the sense of Article 55(1)(c) and therefore may not trigger the reporting obligation.⁵⁷⁶ Since they are part of the relevant information that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. need to keep track of and document, however, the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. will have the possibility to request this information pursuant to Article 91(1). In principle, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are therefore likely not required to report near misses proactively. An exception may arise where a near miss constitutes a ‘reasonable ground’ for updating the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s Safety and Security Framework.⁵⁷⁷ Any such update must be reported to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. within five business days.⁵⁷⁸

2.1.3.3. Keeping track of relevant information

187To be able to document and report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. will need to keep track of relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . The AI Act does not specify how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. should fulfil their obligation to keep track of relevant information. The Safety and Security Chapter of the GPAI Code of Practice, however, lists a number of exemplary measures that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can adopt in order to comply with the obligation.⁵⁷⁹ 

188Signatories may comply with this obligation through a range of measures. On the input side, these measures include collecting end-user feedback, providing reporting channels (including anonymous ones) and incident reporting forms, and offering bug bounties. Broader engagement mechanisms encompass community-driven model evaluations and public leaderboards; frequent dialogues with affected stakeholders; and collaboration with academia, civil society, regulators, and independent researchers in support of the scientific study of the model’s capabilities, propensities, affordances, and effects. On the monitoring side, signatories may monitor software repositories, known malware, public forums, and/or social media for patterns of use; implement privacy-preserving logging and metadata analysis of the model’s inputs and outputs using, for example, watermarks, metadata, and/or state-of-the-art provenance techniques; collect relevant information about breaches of the model’s use restrictions and any subsequent incidents; and monitor aspects of the model that are relevant to assessing and mitigating systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. but are not transparent to third parties, such as hidden chains-of-thought in models whose parameters are not publicly available for download.⁵⁸⁰

189According to Measure 9.1 of the Safety and Security Chapter of the GPAI Code of Practice, signatories are required to additionally ‘review other sources of information (such as police and media reports, posts on social media, research papers, and incident databases)’ and also ‘facilitate the reporting of relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. by downstream modifiers, downstream providers Article 3(68) AI Act: ‘downstream provider’ means a provider of an AI system, including a general-purpose AI system, which integrates an AI model, regardless of whether the AI model is provided by themselves and vertically integrated or provided by another entity based on contractual relations. , users and other third parties’ by informing them of direct reporting channels.

190Lastly, under Measure 8.3 of the Safety and Security Chapter of the GPAI Code of Practice, signatories commit to promote a healthy risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. culture – one indicator for that being that internal reporting channels are actively used and reports are acted upon appropriately.

2.1.3.4. Documenting relevant information

191The AI Act does not specify how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must document the relevant information. Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should at least document the information mentioned in Measure 9.2 of the Safety and Security Chapter of the GPAI Code of Practice. It is not clear whether providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should document more than what they share in their report to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. . Since Measure 9.2 sets out only a minimum standard (‘at least the following information’), it may be advisable for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to document more than the required minimum to the extent that additional relevant information concerning serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. could also inform the assessment of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model.⁵⁸¹ Whether the documentation of additional information is appropriate will have to be determined case by case.⁵⁸²

192The GPAI Code of Practice recommends that signatories keep their documentation for five years after the date of documentation or the date of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , whichever is later.⁵⁸³

2.1.3.5. Possible corrective measures

193 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must keep track of, document and report not only relevant information on serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , but also possible corrective measures. The term is not further defined. The GPAI Code of Practice indicates a broad understanding encompassing all measures aimed to rectify the harm.⁵⁸⁴ Drawing on Article 2(67) MDR, it can be understood to mean any possible action that can be taken to eliminate the cause of the serious accident as well as the incident itself and its consequences. Possible corrective measures will often include changes to the model itself as well as notices to downstream AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁵⁸⁵ Additionally, the information gathered on a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. as well as the corrective measures taken will inform the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. estimation, which in turn influences what (further) safety mitigations pursuant to Article 55(1)(b) should be introduced.⁵⁸⁶ It can therefore be the case that corrective measures adopted ex post in response to a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. become safety mitigations ex ante for compliance with Article 55(1)(b).

2.1.3.6. Reporting relevant information and possible corrective measures

194 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must ‘report, without undue delay, to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and, as appropriate, the national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. ’.⁵⁸⁷ This section will deal with each element of the definition in turn.

2.1.3.6.1. Without undue delay

195Unlike Article 73, Article 55(1)(c) contains no further specifications or differentiated timelines for reporting different types of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . Instead, the provision refers only to the indeterminate standard of ‘without undue delay’. Accordingly, an individual assessment must, in principle, be made in each specific case.

196Some authors suggest that guidance can be drawn from the timelines mentioned in Article 73, as both reporting obligations pursue similar objectives.⁵⁸⁸ According to those timelines, serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. should, in principle, be reported ‘immediately after the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. has established a causal link between the AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. and the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. or the reasonable likelihood of such a link, and, in any event, not later than 15 days after the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. or, where applicable, the deployer Article 3(4) AI Act: ‘deployer’ means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. , becomes aware of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’,⁵⁸⁹ while also taking into account the severity of the incident.⁵⁹⁰ The latter point is further specified in the subsequent paragraphs of Article 73. So-called widespread infringements Article 3(61) AI Act: ‘widespread infringement’ means any act or omission contrary to Union law protecting the interest of individuals, which: (a) has harmed or is likely to harm the collective interests of individuals residing in at least two Member States other than the Member State in which: (i) the act or omission originated or took place; (ii) the provider concerned, or, where applicable, its authorised representative is located or established; or (iii) the deployer is established, when the infringement is committed by the deployer; (b) has caused, causes or is likely to cause harm to the collective interests of individuals and has common features, including the same unlawful practice or the same interest being infringed, and is occurring concurrently, committed by the same operator, in at least three Member States. – as defined in Article 3(61) – as well as serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. as defined in Article 3(49)(b) (that is, serious and irreversible disruptions of the management or operation of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. ) must be reported immediately, and not later than two days after the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. (or deployer Article 3(4) AI Act: ‘deployer’ means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. ) becomes aware of the incident).⁵⁹¹ In the event of the death of a person, the report must be provided immediately, and no later than ten days after the date on which the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. (or deployer Article 3(4) AI Act: ‘deployer’ means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. ) becomes aware of the incident.⁵⁹²

197Although this approach offers the advantage of giving providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. clear timelines to adhere to, it nonetheless appears unconvincing at first glance to blindly transfer Article 73’s timelines to Article 55(1)(c).⁵⁹³ Although the reporting obligations in Article 73 and Article 55(1)(c) are similar, and both rely on the same or at least a similar concept of ‘ serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. ’ as a key element, the Union legislature would likely have incorporated the Article 73 timelines directly or by explicit cross-reference if they had wanted to apply them to Article 55(1)(c).

198Nevertheless, the GPAI Code of Practice works with similar graduated timelines and also further distinguishes between different types of reports – an initial report, an intermediate report and a final report.⁵⁹⁴ This structure closely resembles the distinction found in the NIS2 Directive (Article 23(4)) but is not explicitly found in Article 55(1)(c). According to the GPAI Code of Practice, an initial report⁵⁹⁵ is to be provided by the signatories at the following times ‘if the involvement of their model (directly or indirectly) led to:’⁵⁹⁶

‘a serious and irreversible disruption of the management or operation of critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. , or if the Signatories establish or suspect with reasonable likelihood such a causal relationship between their model and the disruption, not later than two days after the Signatories become aware of the involvement of their model in the incident;’⁵⁹⁷

‘a serious cybersecurity breach, including the (self-)exfiltration of model weights and cyberattacks, or if the Signatories establish or suspect with reasonable likelihood such a causal relationship between their model and the breach, not later than five days after the Signatories become aware of the involvement of their model in the incident;’⁵⁹⁸

‘a death of a person, or if the Signatories establish or suspect with reasonable likelihood such a causal relationship between their model and the death, not later than 10 days after the Signatories become aware of the involvement of their model in the incident;’⁵⁹⁹ 

‘serious harm to a person’s health (mental and/or physical), an infringement of obligations under Union law intended to protect fundamental rights, and/or serious harm to property or the environment, or if the Signatories establish or suspect with reasonable likelihood such a causal relationship between their model and the harms or infringements, not later than 15 days after the Signatories become aware of the involvement of their model in the incident.’⁶⁰⁰

199According to the GPAI Code of Practice, signatories are then required – in cases of unresolved serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. – to update the information provided and add the further information required⁶⁰¹ in an intermediate report.⁶⁰² Such an intermediate report shall be provided every four weeks after the initial report.⁶⁰³ The final report must then be provided not later than 60 days after the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. has been resolved.⁶⁰⁴ This final report should then cover all the information discussed above.⁶⁰⁵

200Even though, as noted above, one should not blindly transfer the timelines given in Article 73, there are compelling reasons in favour of a graduated approach as in the GPAI Code of Practice. Given the purpose of the obligation – to enable a coordinated response to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and thereby secure the situation after an incident, restore capacity to act, and prevent further harm – it indeed makes sense to report certain pieces of information to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. at different points in time as soon as they become available, rather than waiting until all relevant information has been gathered. The AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. can already work with the information contained in the initial report and, where appropriate, take first measures in response to the incident.⁶⁰⁶ Moreover, the wording of the provision can be read accordingly such that ‘without undue delay’ refers not the reports as such but to the (respective) relevant information. It can thus reasonably be understood to mean that different timelines may be appropriate for different pieces of information. In that case, it would not be problematic that a graduated approach is not expressly set out, as it is in Article 73 AI Act or Article 23(4) NIS2 Directive.

201A remaining difficulty concerns the point in time from which the assessment must begin as to whether a report has been made ‘without undue delay’. Possible reference points include, on the one hand, the moment of first suspicion, or the point at which the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. is sufficiently certain (depending on the interpretation followed above),⁶⁰⁷ that the involvement or an incident or malfunction of its model (directly or indirectly) caused one of the outcomes specified in Article 3(49) or a material increase or the materialisation of another systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . On the other hand, it could be argued that the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must have actually established the causal relationship. The latter view is supported by the fact that the obligation in Article 55(1)(c) presupposes the existence of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , which in turn – by its definition – requires the necessary causal relationship.⁶⁰⁸ If one were to strictly adhere to this wording, a report would therefore always have to be submitted only after the causal relationship has been established – albeit then ‘without undue delay’.

202This interpretation is also indicated in the GPAI Code of Practice, stating that signatories should report not later than two days after they ‘become aware of the involvement of their model in the incident’⁶⁰⁹ – also presupposing an actual incident. In light of the purpose of the reporting obligation, however, namely to enable a prompt and coordinated response to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. , it seems more appropriate to assume a reporting duty as soon as the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. suspects a causal relationship. It could be countered that the Union legislature explicitly allows such suspicion to suffice only in Article 73(4), which might suggest, a contrario, that suspicion should not suffice elsewhere. Yet, it appears more likely that – in both Articles 73 and 55(1)(c) – the amendment of the Article 3(49) definition from ‘leads, might have led or might lead’ to ‘leads’ in the final version was simply not taken into account.⁶¹⁰ Under the earlier definition, the suspected causal relationship would already have been included in the definition of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. should therefore submit a report ‘without undue delay’ from the time they suspect a causal connection, not only once they have positively established it.

2.1.3.6.2. AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission.

203 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. must report ‘to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and, as appropriate, to national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. ’.⁶¹¹ According to Article 55(3), the information or documentation obtained pursuant to Article 55(1)(c) must be treated confidential in the sense of Article 78.⁶¹²

204The Commission has published a reporting template for serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. involving general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .⁶¹³  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should use the EU SEND platform to fulfil their reporting obligation under Article 55(1)(c). The Commission has published technical guidance for submitting documents via EU SEND on its website.⁶¹⁴

205Article 55(1)(c) does not clarify whether – and, if so, which – follow-up obligations of the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. exist.⁶¹⁵ By contrast, the NIS2 Directive expressly provides that the ‘CSIRT or the competent authority shall provide, without undue delay and where possible within 24 hours of receiving the early warning referred to in paragraph 4, point (a), a response to the notifying entity, including initial feedback on the significant incident and, upon request of the entity, guidance or operational advice on the implementation of possible mitigation measures’.⁶¹⁶ Accordingly there seems to be no institutional duty for the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. to provide reporting providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. with guidance or operational advice. Nor is it expressly regulated whether (potentially) affected persons and/or the public must be informed of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. . Comparable provisions on informing the public can be found, for example, in Article 23(7) NIS2 Directive or Article 17(2) CRA.

2.1.3.6.3. National competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor.

206Article 55 does not clarify when exactly it is appropriate to report the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. . It is likely that this will be the case when their jurisdiction is triggered, in particular due to an incident that has effects within the Member State.⁶¹⁷ This could, for instance, be the case where citizens are killed or injured, where the state’s critical infrastructure Article 3(62) AI Act: ‘critical infrastructure’ means critical infrastructure as defined in Article 2, point (4), of Directive (EU) 2022/2557. is being disrupted, or where property within the Member State is affected. Similarly, this could be the case where a cyber threat originates from within the Member State’s territory.

2.1.3.6.4. Reporting does not entail admission of wrongdoing

207Recital (j) of the Code of Practice states that ‘[t]he Signatories recognise that the reporting of a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. is not an admission of wrongdoing’.⁶¹⁸ As a preliminary matter, the normative weight of this formulation must be assessed with care. On its plain wording, recital (j) addresses only the evidentiary status of the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s report: it shields the act of reporting from being construed as an acknowledgement of fault but says nothing about the underlying conduct giving rise to the incident. The mere fact that reporting does not constitute an admission of wrongdoing does not preclude the existence of wrongdoing itself. Accordingly, the formulation cannot be read as precluding the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. from characterising the conduct underlying a reported incident as wrongdoing – let alone as conferring a general liability exemption on reporting providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .

208This reading is reinforced by comparison with provisions in EU law that unambiguously establish liability exemptions through explicit language. The safe harbour provision in the DSA⁶¹⁹ – Article 6 – states in unequivocal terms that ‘the service provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. shall not be liable’. Equally explicit is the Regulation on the reporting, analysis and follow-up of occurrences in civil aviation,⁶²⁰ which provides that ‘[t]he sole objective of occurrence reporting is the prevention of accidents and incidents and not to attribute blame or liability’.⁶²¹ Against this background, the formulation in the Code of Practice falls short of the standard of clarity that EU law seems to demand of a genuine liability exemption.⁶²²

209It might be objected that reading a liability exemption into recital (j) could be normatively desirable. Such an exemption would come with distinct policy advantages: fear of liability – alongside broader legal uncertainty – constitutes a disincentive for companies to report incidents,⁶²³ and this concern gains particular force with respect to incidents that are unlikely to be externally detectable – for example incidents that do not result in a materialisation of harm but only materially increase the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model.⁶²⁴ In the absence of robust reporting incentives – other than existing fines – such incidents may go unreported entirely, leaving the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. uninformed. While this is a weighty consideration,⁶²⁵ it cannot override the plain meaning of the text and would be more appropriately addressed through legislative intervention than interpretive extension.

210It follows that the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. retains full authority to initiate investigations in response to a report – including investigations into whether the reporting provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. has taken adequate measures to mitigate the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by its model. It might well be the case that the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. has not violated its obligations under Article 55(1)(a) and (b), and yet a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. still occurs. That said, a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s consistent adherence to its reporting obligations may be treated as evidence of its broader commitment to regulatory compliance, a factor the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. can take into account when exercising its discretion to impose fines under Article 101,⁶²⁶ in particular in light of the principle of proportionality. Moreover, one may read into Article 101 the principle underlying Article 99(7)(h),⁶²⁷ pursuant to which particular account shall be taken of the extent to which the operator Article 3(8) AI Act: ‘operator’ means a provider, product manufacturer, deployer, authorised representative, importer or distributor. notified the infringement on its own initiative. Reporting compliance does not therefore operate as a liability exemption – though it is not necessarily without legal consequence either.

2.1.3.7. Location of the incident

211It is, at first glance, unclear whether the obligation to document, keep track of and report relevant information also covers serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. that occur outside the European Union. This question is quite important since many providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are established in third countries. Three questions must be distinguished.⁶²⁸ The first is whether the EU legislature can regulate conduct outside of the Union. The second is whether the EU legislature has actually exercised that possibility in Article 55(1)(c), read in light of the provision’s wording, its systematic context within the AI Act, and its regulatory purpose.⁶²⁹ Third, it must be assessed whether any limits or exceptions speak against extending the obligation under Article 55(1)(c) to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. happening outside the European Union.

2.1.3.7.1. Existence of extraterritorial jurisdiction triggers

212In general, the Court of Justice of the European Union (“CJEU”) permits territorial extensions of EU legislation on the basis of one of seven triggers: nationality, presence, conduct, (qualified) effects, anti-evasion, counterparty and property.⁶³⁰ For the question at hand, the conduct trigger is of particular relevance. Additionally, the qualified effects trigger might support a reading under which Article 55(1)(c) extends to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. happening outside the European Union.

213The conduct trigger ‘relates to an activity that at least partly occurs in the EU’s territory’ and requires ‘some activity of a foreign entity […] to connect to the EU’s territory’.⁶³¹ A variant of the conduct trigger – market access – appears most relevant for the purposes of the present analysis.⁶³² The notion that the EU legislature may condition access to its market on compliance with EU standards has also been recognised by the CJEU. In United Airlines,⁶³³ the Court held that ‘the EU legislature may in principle choose to permit a commercial activity […] to be carried out in the territory of the European Union only on condition that operators Article 3(8) AI Act: ‘operator’ means a provider, product manufacturer, deployer, authorised representative, importer or distributor. comply with the criteria that have been established by the European Union and are designed to fulfil the […] objectives which it has set for itself’.⁶³⁴ 

214With regard to the question at hand, the conduct trigger is satisfied where a GPAI model provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. places its model on the EU market. By doing so, GPAI model providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. subject Article 3(58) AI Act: ‘subject’, for the purpose of real-world testing, means a natural person who participates in testing in real-world conditions. themselves to the rules of the AI Act, which, under Article 2(1)(a), expressly cover GPAI model providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ‘irrespective of whether those providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are established or located within the Union or in a third country’.⁶³⁵

215The qualified effects trigger – which the CJEU has recently held ‘may, on its own, serve as the basis for the Commission’s jurisdiction’⁶³⁶ – is another basis for the extraterritorial application of EU law that may be able to offer additional support for a reading under which Article 55(1)(c) also extends to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. happening outside the European Union. The qualified effects doctrine, developed in competition law and merger control,⁶³⁷ captures conduct outside the Union that produces foreseeable, immediate and substantial effects on the internal (Union) market.⁶³⁸ While it falls to the Commission to demonstrate that conduct has foreseeable, immediate and substantial effects in the European Union,⁶³⁹ it seems possible to argue that it is foreseeable placing models that present systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. on the Union market will have immediate and substantial effects on the internal Union market.

2.1.3.7.2. Use of extraterritorial jurisdiction triggers

216As stated above, the mere possibility of extraterritorial regulation does not, in itself, imply that the EU legislature intended a particular provision to apply extraterritorially. Rather, determining whether a provision carries extraterritorial reach requires a comprehensive interpretation of both the provision at issue and its systematic context.⁶⁴⁰ The CJEU implicitly adopted this approach in Google v CNIL,⁶⁴¹ contending that although the purpose and objectives of an EU instrument may in principle justify its extraterritorial application,⁶⁴² it must also be apparent from the provisions at issue that the EU legislature ‘ha[d] chosen to confer a scope on the rights enshrined in those provisions which would go beyond the territory of the Member States’.⁶⁴³ Following this approach, this subsection now turns to whether Article 55(1)(c) should be interpreted as extending to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. occurring outside the Union, having regard to the broader objective of Article 55 to assess and mitigate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

217The obligation under Article 55(1)(c) to keep track of, document and report relevant information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. should be understood as further specifying the overarching obligation under Article 55(1)(b) to assess and mitigate systemic risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .⁶⁴⁴  Systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. is defined in Article 3(65) as a risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. ‘having a significant impact on the Union market […] due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain’. Thus, the ability to propagate negative effects at scale is identified as an essential characteristic of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .⁶⁴⁵ In particular, propagation occurs across the AI value chain, which in turn is understood broadly to span from GPAI model providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. deployers Article 3(4) AI Act: ‘deployer’ means a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity. and end users,⁶⁴⁶ including actors who need not be based in the EU, provided the model is placed on the Union market. This understanding is reflected in Recital 110, which describes systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. in terms of an event giving rise to ‘a chain reaction with considerable negative effects that could affect up to an entire city, an entire domain activity or an entire community’.⁶⁴⁷

218The structure of Article 3(65) further supports this reading. The provision defines systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. as ‘a risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. having a significant impact on the Union market’, which may arise by virtue of ‘actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole’.⁶⁴⁸ This indicates that harm to ‘society as a whole’ operates as one of the pathways through which a significant impact on the Union market may be established. Therefore, importantly, ‘the society as a whole’ criterion could be satisfied by negative effects that materialise outside Union territory,⁶⁴⁹ provided a significant impact is realised on the Union market as result of those negative societal effects. A GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that produces negative effects outside the EU but causes harm at a societal level is therefore not outside the AI Act’s regulatory scope by virtue of its location, provided those negative effects are reasonably foreseeable to propagate across the value chain and impact the Union market.

219Indeed, Article 55(1)(b) requires that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model throughout its development, placing on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , and use – that is, along its lifecycle.⁶⁵⁰ Notably, model development, including training and fine-tuning, may take place outside the EU even for GPAI models that are placed on the EU market, for example if computing infrastructure is located abroad. It may therefore be inferred that the intention of the EU legislature was to ensure that all processes pertaining to risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation regardless of the geographical location of those processes, and which ultimately determine whether the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. posed by a model are acceptable for the model to be placed on and be used in the Union market, fall within the scope of the obligation under Article 55.⁶⁵¹ This logic reflects traditional EU product safety, which requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ensure that their product complies with EU standards prior to market placement in a manner that effectively produces extraterritorial effects.⁶⁵²

220As such, the obligations imposed on providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. under Article 55 must be interpreted and understood in light of the EU legislature’s recognition that systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may arise at any stage of the GPAI model’s lifecycle,⁶⁵³ including at stages taking place outside the EU, and must nonetheless be assessed and mitigated. More specifically, Article 55(1)(a) and (b) make clear that the obligations to assess and mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. apply across the entire lifecycle of a GPAI model.

221Such a reading is also functionally necessary in light of the objective of Article 55(1), namely to ensure appropriate systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation. Serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. serve as a key trigger for the reassessment of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. acceptance determination.⁶⁵⁴ Excluding serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. on the basis of their location would undermine this function and contradict the requirement that risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation measures account for the entire model lifecycle.⁶⁵⁵ This is also reinforced by an analysis of the serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. concept specific to Article 55(1)(c). As noted above, the concept of serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. in Article 55(1)(c) is likely to be understood more broadly than the definition in Article 3(49) suggests.⁶⁵⁶ A territorial restriction that required a serious incident Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to occur on Union territory would give rise to difficulties on several counts for the application of Article 55(1)(c). Again, as noted above, whilst the GPAI Code of Practice formulates the reporting trigger broadly – seemingly requiring only that the GPAI model be involved in, for example, a serious cybersecurity breach – the Commission’s GPAI Guidelines are more specific, clarifying that a breach ‘related to’ a GPAI model suffices.⁶⁵⁷ In other words, it seems like some of the serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. covered by Article 55(1)(c) need not originate from the GPAI model; it can be sufficient that they happen to it.⁶⁵⁸

222Were one to assume that only serious cybersecurity breaches – and, depending on the interpretation followed above, other events that cause a material increase in systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. posed by the model⁶⁵⁹ – occurring within the Union are covered, the obligation to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. under Article 55(1)(c) would largely be deprived of its practical effect. The physical infrastructure and the training and development activities related to GPAI models that pose systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. themselves will, in the overwhelming majority of cases, be located outside the Union. Moreover, it will often prove impossible to identify the territorial origin of cyberattacks on the model. To prevent the obligation in Article 55(1)(c) from being rendered ineffective with respect to serious cybersecurity breaches, such breaches occurring outside the Union to a GPAI model placed on the Union market should also fall within its scope.

2.1.3.7.3. Limits and exceptions

223Lastly, interpreting Article 55(1)(c) as to encompass serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. occurring outside the European Union is not unreasonable.⁶⁶⁰ From a policy perspective, the extraterritorial application of EU law can be seen as ‘unreasonable when a state with weaker interests exercises prescriptive extraterritorial jurisdiction and does not defer to the state with stronger interests’.⁶⁶¹ To explore and explain this doctrine, one can refer to the Google Spain⁶⁶² and Google v CNIL cases.⁶⁶³ In both, the CJEU sought to reconcile the extraterritorial application of EU law with other interests, in particular those of third states.⁶⁶⁴ Particularly in Google v CNIL, the Court recognised that third states hold the higher interest compared to the EU in determining the circumstances under which content should be de-referenced from search engines within their territories, particularly when balancing privacy rights against freedom of information.

224For Article 55(1)(c), however, those criteria are arguably not fulfilled. What is at stake in Article 55(1)(c) is not an order to de-reference content – that is, to actively interfere with the public availability of content in a third state – but rather the extension of a reporting obligation concerning serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to incidents occurring outside the Union. Whereas de-referencing directly removes content from search results in a third state’s web version, thereby displacing that state’s own calibration of freedom of information versus privacy rights, the reporting obligation at issue in Article 55(1)(c) amounts to, at worst, a duplication of the reporting obligation: the third state remains free to determine when it requires the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to report serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. to its competent authorities. Parallel reporting thus does not override the third state’s regulatory interests in the same manner as an extraterritorial de-referencing order.⁶⁶⁵ It therefore strongly follows that the obligation under Article 55(1)(c) should be considered to extend to serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. occurring outside the Union as well.

2.1.4. Article 55(1)(d): Cybersecurity protection

225Article 55(1)(d) requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ensure an adequate level of cybersecurity protection for both the GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and the physical infrastructure of the model.

2.1.4.1. General remarks

226Article 55(1)(d) serves the purpose of ensuring the cybersecurity of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and thereby a consistent and high level of protection of public interests throughout the Union.⁶⁶⁶ Additionally, the rationale of the provision can be seen in promoting the trustworthiness of AI in general,⁶⁶⁷ which in turn is aimed at creating a safe and innovation-friendly environment. The provision can also be read in light of the EU’s cybersecurity agenda.⁶⁶⁸ The widespread deployment of AI gives rise to new and AI-specific risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. which traditional approaches to cybersecurity may be unable to address adequately.⁶⁶⁹ Ensuring a sufficient Union-wide level of cybersecurity is described as ‘one of the key challenges for the Union’ and is essential for strengthening both the Union’s economy and democracy.⁶⁷⁰

227As discussed above, the constituent obligations under Article 55(1) ‘complement and feed into each other.’⁶⁷¹ Against this background, the relationship between Article 55(1)(d) – the focus of this section – and Article 55(1)(b), which requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to ‘assess and mitigate possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. at Union level’, requires further clarification, particularly as regards the notion of cybersecurity risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .⁶⁷² 

228Article 55(1)(b) primarily captures possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. that emanate from the GPAI model itself as opposed to risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to the model.⁶⁷³ As such, this relates, in particular, to cyber-offensive capabilities that scale with the model’s capabilities and are explicitly identified by the Safety and Security Chapter of the GPAI Code of Practice as a specified category of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .⁶⁷⁴ Article 55(1)(d), however, appears to pursue a broader objective: it seeks to ensure an adequate level of cybersecurity even beyond those cases in which vulnerabilities themselves qualify as possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , since cybersecurity vulnerabilities may act as triggers for, or amplifiers of, pre-existing systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. factors in high-impact models.⁶⁷⁵ For example, even the most robust safety mitigations may prove ineffective if the model is infiltrated or its parameters are illegitimately copied as this could allow a similar systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. to present itself elsewhere. Article 55(1)(d) thus serves to ensure that measures taken pursuant to Article 55(1)(a) and (b) are – and remain – effective. It thereby tries to ‘limit the scenarios that could lead to materialised systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.⁶⁷⁶ In other words, Article 55(1)(d) mainly focuses on model and infrastructural integrity in general – risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to the model – thereby securing effective risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. assessment and mitigation, while Article 55(1)(b) focuses on possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. emanating from the model.

229The cybersecurity duties imposed on GPAI models under Article 55 and high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 15 should not be read in isolation.⁶⁷⁷ They operate within a wider framework of EU laws on cybersecurity, including the Cyber Resilience Act⁶⁷⁸, the Cybersecurity Act (“CSA”)⁶⁷⁹ and the NIS2 Directive.⁶⁸⁰ For non-GPAI models as well as GPAI models without systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and non-high-risk systems, the AI Act does not impose any explicit additional cybersecurity requirements,⁶⁸¹ instead deferring to the general cybersecurity framework set out in the mentioned legislations.

2.1.4.2. Meaning of cybersecurity

230The AI Act does not provide a definition for cybersecurity as used in Article 55(1)(d) (or Article 15).⁶⁸² The key interpretive question in this regard is how narrow or broad the term should be understood, as this informs the breadth of the providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’ obligations. Three approaches may aid in a better understanding of the meaning of the term cybersecurity in Article 55(1)(d). First, one could draw a comparison to the cybersecurity obligation for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. under Article 15. Second, guidance might be found in looking at definitions of cybersecurity in other EU legal acts the AI Act refers to. Third, the GPAI Code of Practice and, particularly, its security measures may help shed light on the meaning of cybersecurity in Article 55(1)(d). Importantly, these approaches should not be seen as alternatives to each other, rather their respective insights, taken together, aid in developing a workable understanding of the term in Article 55(1)(d).

2.1.4.3. Approaches to define cybersecurity

2.1.4.3.1. Comparison to Article 15

231Article 15 lays down requirements on the ‘[a]ccuracy, robustness and cybersecurity’ for high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. . The understanding of ‘cybersecurity’ under that provision could help elucidate the meaning of cybersecurity in the context of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. .

232According to Article 15(1), high-risk systems ‘shall be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle’. Article 15(5) further concretises the cybersecurity requirements. Its first sentence stipulates that high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ‘shall be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities’. While this clearly does not amount to a definition of cybersecurity, it provides interpretive guidance as to the scope attributed to the concept in Article 15.

233It seems like this understanding in Article 15 is limited in several ways. First, it is noteworthy that Article 15(5) covers resilience only ‘against attempts by unauthorised third parties’ (emphasis added). Similarly, Recital 76 refers to ‘malicious third parties exploiting the system’s vulnerabilities’. The AI Act does not provide for a more detailed definition of what constitutes a third party.⁶⁸³ At a minimum, however, the term arguably encompasses all persons outside of the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s organisation who are not authorised to access or use the system.⁶⁸⁴ Conversely, individuals acting under the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s authority – such as employees – do not appear to qualify as third parties.⁶⁸⁵ As a result, Article 15(5) does not address insider threats – that is, threats arising from individuals from within the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s organisation who misuse their authorised access in order to harm the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁶⁸⁶ Moreover, Article 15(5) first sentence covers only attempts ‘to alter [the high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ] use, outputs or performance’ (emphasis added). By implication, malicious attempts that are not directed at one of these three outcomes fall outside the scope of Article 15(5) – for example, the unlawful extraction of data through other channels than the system’s output.⁶⁸⁷ Finally, Article 15(5)’s first sentence refers exclusively to the exploitation of ‘system vulnerabilities’ (emphasis added).⁶⁸⁸ This, again, leaves insider threats unaddressed. Where an insider exploits authorised access to a system that might otherwise be free of vulnerabilities, such conduct would not fall within the scope of Article 15(5), first sentence.⁶⁸⁹ 

234Article 15(5), third sentence, then turns to ‘AI specific vulnerabilities’. This sentence seems to be understood as a concretisation of the first sentence, with AI-specific vulnerabilities treated as a subcategory of system vulnerabilities.⁶⁹⁰ It provides that the technical solutions adopted to address those risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. shall, where appropriate, include ‘measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data Article 3(29) AI Act: ‘training data’ means data used for training an AI system through fitting its learnable parameters. set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws’. Notably, Article 15(5)’s third sentence introduces requirements that directly target the underlying model – an atypical approach for a provision in Chapter III.⁶⁹¹ Where the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. is also the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of the underlying model, compliance with these requirements is unlikely to raise structural difficulties. This situation may differ, however, where the high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. builds upon a third-party model. This raises the question of the interaction with Article 55(1)(d), in particular as regards the allocation of cybersecurity-related obligations between high-risk AI system Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. and providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . In this regard, interfaces controlled by the system provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will generally fall within that actor’s realm of responsibility.⁶⁹² Where interfaces are instead controlled by the model provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , the model provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. is required to draw up and make available to the system provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. the information and documentation necessary to enable the latter to comply with its obligations, in accordance with Article 53(1)(b).

235Finally, Article 15(4) suggests that cybersecurity, within the meaning of Article 15, is primarily concerned with intentional causes, while unintentional causes fall more naturally under the concept of robustness in Article 15(4).⁶⁹³ It should be noted, however, that the two concepts do not lend themselves to a fully clean distinction. The choice to place both within the same provision of the AI Act appears to be a deliberate one, indicating their close relationship. Particularly in light of Article 15(5) last sentence, also addressing confidentiality attacks, cybersecurity may implicitly encompass certain robustness dimensions. Indeed, from a technical standpoint, some authors take the view that adversarial robustness is but one aspect of cybersecurity.⁶⁹⁴

236In summary, the concept of cybersecurity underlying Article 15 appears to be comparatively⁶⁹⁵ narrow. This is because it is confined, first, to attempts by unauthorised third parties, thereby excluding insider threats. Furthermore, Article 15 primarily seems to cover only those attempts aiming at altering the use, output or performance of a system with other malicious inferences falling outside the scope. Finally, Article 15(5) mainly addresses intentional causes, whereas unintentional causes are dealt with under the notion of robustness in Article 15(4).

2.1.4.3.2. References to other instruments defining cybersecurity

237The AI Act also refers to other legal instruments – namely the CSA and the CRA – with regard to its cybersecurity requirements. These interconnections could further help guide the interpretation of the concept in the AI Act in general and Article 55(1)(d) respectively.

238According to Article 15, read in conjunction with Article 42(2), high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. ‘that have been certified or for which a statement of conformity has been issued under a cybersecurity scheme pursuant to [the CSA] and the references of which have been published in the Official Journal of the European Union shall be presumed to comply with the cybersecurity requirements set out in Article 15 of this Regulation in so far as the cybersecurity certificate or statement of conformity or parts thereof cover those requirements’. Some authors argue that this indicates that the definition of cybersecurity found in the CSA also applies to the AI Act.⁶⁹⁶ 

239That definition found in the CSA is broad, encompassing all ‘activities necessary to protect network and information systems, the users of such systems, and other persons affected by cyberthreats’. Article 2(1) CSA defines cyberthreats as ‘any potential circumstance, event or action that could damage, disrupt or otherwise adversely impact network and information systems, the users of such systems and other persons’. This formulation – and thus the CSA’s definition of cybersecurity – contrasts the characteristics distilled from the analysis of Article 15 AI Act in some points.⁶⁹⁷ First, the broad definition contained in the CSA is not limited to attacks by third parties and therefore also encompasses insider threats. Second, it is not confined to attempts aiming at altering the use, output or performance of a system, but instead covers any adverse impact on networks and information systems, as well as users and other persons. Third, this definition is not restricted to intentional causes; it also includes unintentional causes, which, under Article 15 AI Act, are treated as matters of robustness rather than cybersecurity.

240One argument against adopting the broad understanding of the term cybersecurity as defined in the CSA to Article 55(1)(d) is that no provision equivalent to Article 42(2) AI Act exists for GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Even if one were to argue that Article 42(2) read in conjunction with the CSA provides the basis for a broad understanding of the term cybersecurity in Article 15, this reasoning would not automatically extend to GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. given the absence of an explicit corresponding cross-reference. Moreover, Article 42(2) expressly refers to Article 15 only. Had the EU legislature intended to rely on the CSA’s definition throughout the Act, it would have made more sense to refer to that definition expressly. Additionally, even if one were to argue that the same must apply to GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. , it seems like there would be no big incentive for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to obtain such a certification because providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. will, in practice, most likely rather rely on the Code of Practice to demonstrate compliance with Article 55(1)(d).⁶⁹⁸ 

241Another relevant EU legal act addressing cybersecurity is the CRA, which the AI Act refers to in its Recital 77.⁶⁹⁹ High-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. falling within the scope of the CRA ‘may demonstrate compliance with the cybersecurity requirements of [the AI Act] by fulfilling the essential cybersecurity requirements set out in [the CRA]’. The counterpart to the AI Act’s Recital 77 can be found in Article 12 CRA on high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. .⁷⁰⁰ This sets out the exact requirements for when compliance with the CRA results in high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. being ‘deemed to comply with the cybersecurity requirements set out in Article 15 [of the AI Act]’. Since the CRA refers to the CSA for the definition of cybersecurity,⁷⁰¹ Article 12 CRA also appears to be based on the broad understanding of the term cybersecurity described above.⁷⁰² Once again, however, an argument against extending this broad concept of cybersecurity to Article 55(1)(d) is that legal concepts can be understood relatively – even within a single piece of legislation – and that, in the absence of a provision identical to the Article 12 CRA provision for GPAI models (with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ),⁷⁰³ it appears that the legislature did not intend harmonisation of the concept within the scope of Article 55(1)(d).⁷⁰⁴

242In summary, the cross-references within the AI Act suggest an endorsement of a broad conception of cybersecurity in Article 55(1)(d), even if this sits uneasily with the comparatively narrow understanding reflected in Article 15(5). Although these provisions are expressly confined to high-risk AI systems Article 3(1) AI Act: ‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments. , and there are arguments against a direct adoption of the concept to Article 55(1)(d), they might still be of relevance for the interpretation of Article 55(1)(d), since they seem to offer the most developed picture of what cybersecurity means within the AI Act. This cannot determine, but it at least informs, the interpretation of the concept’s meaning in Article 55(1)(d).

2.1.4.3.3. Code of Practice

243Further guidance for the understanding of the concept of cybersecurity in Article 55(1)(d) can be drawn from the Safety and Security Chapter of the GPAI Code of Practice,⁷⁰⁵ as its security commitments help indicate the scope of ‘cybersecurity’ in Article 55(1)(d). Although the Code itself is not legally binding on non-signatories,⁷⁰⁶ it can offer interpretive value in the form of ‘expert‑crafted guidance’.⁷⁰⁷ At the moment, the GPAI Code of Practice is therefore a key point of guidance in informing all providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’ breadth of obligations under Article 55(1)(d). That is why the following paragraphs will examine the Code’s requirements in detail, with a view to distil insights on the scope and meaning of cybersecurity in Article 55(1)(d).

244As the Commission’s adequacy assessment rightly finds, ‘Commitment 6 specifies how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. may ensure an adequate level of cybersecurity protection … pursuant to Article 55(1), point (d).’⁷⁰⁸ To this end, the commitment requires signatories to define a Security Goal identifying the threat actors their mitigations are designed to address – including non-state external threats, insider threats, and other anticipated threat actors. The aforementioned Security Goal can be met by signatories by implementing appropriate security mitigations – staged appropriately in line with the increase in model capabilities.⁷⁰⁹ Appendix 4 to the Code of Practice specifies these. Signatories deviating from the mitigations mentioned in Appendix 4 will need to implement alternative adequate mitigations, which the European Commission will likely assess in reference to the ones listed.⁷¹⁰ The following paragraphs will deal with different components to the cybersecurity requirements in Article 55(1)(d), which are evident from the categories under the Code of Practice, in turn.

2.1.4.3.3.1. General security mitigations

245A first component of the requirements under Article 55(1)(d) – as reflected in Appendix 4.1 of the Code – concerns general security mitigation measures. These measures aim to prevent unauthorised network access and reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of social engineering, malware infection and malicious use of portable devices, and vulnerability exploitation and malicious code execution.⁷¹¹ All of these mitigation measures are designed to ensure that only authorised persons have access to the model and other sensitive information and that the integrity of the model and its infrastructure is safeguarded. This illustrates the focus of Article 55(1)(d) on security for the model itself, thereby preventing or at least minimising possible systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. emanating from it.

246Unauthorised network access is to be prevented by requiring providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to implement ‘strong identity and access management practices, including restrictions on device and account sharing, multi-factor authentication, strong password enforcement, strong access management tools, 802.1x authentication,⁷¹² zero trust architecture,⁷¹³ protection of wireless networks to the same standard as wired networks,⁷¹⁴ and the separation of any guest networks from the work network’.⁷¹⁵ The GPAI Code of Practice does not, however, prescribe in detail how exactly these measures are to be implemented. Multi-factor authentication (“MFA”), for example, may be realised in various forms. It may rely on knowledge factors (something an individual knows, such as a security question), possession factors (something an individual possesses, such as a physical token), inherent factors (something an individual is or has, such as physical characteristics) or location factors.⁷¹⁶ Not all combinations of these factors offer an equivalent level of security,⁷¹⁷ so providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should still ensure that they do not adopt the measures mentioned in the GPAI Code of Practice in a purely formal manner but instead assess their suitability and security in light of their specific model. Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might find useful guidance in the Commission’s implementing regulation for the application of the NIS2 Directive⁷¹⁸ as well as in ENISA’s accompanying technical implementation guidance, especially with regard to MFA.⁷¹⁹ 

247 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are further required to reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of social engineering⁷²⁰ by implementing ‘email filtering for suspicious attachments, links and other phishing attempts’.⁷²¹ It appears reductive to confine mitigation efforts with regard to social engineering to the filtering of emails while excluding other communication channels like Slack or Teams.⁷²² It is, however, doubtful whether the GPAI Code of Practice can be interpreted to the effect that email filtering is intended to serve merely as an example. In its third draft, the Code of Practice referred more broadly to ‘strong protections against social engineering’⁷²³ – its adopted version, however, only refers to the ‘reduction of risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of social engineering’ and expressly only mentions email filtering.⁷²⁴

248Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of malware infection and of malicious use of portable devices through ‘policies regarding the use of removable media’.⁷²⁵ This can include, for example, the prohibition of connecting removable media without an organisational reason for use or scanning the media for malicious code before use.⁷²⁶ Attackers might even place portable devices in or close to target facilities – for example in a parking lot⁷²⁷ – hoping that employees would plug them into their computer.⁷²⁸ Again, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might find useful guidance in the Commission’s implementing regulation for the application of the NIS2 Directive⁷²⁹ as well as ENISA’s accompanying technical implementation guidance.⁷³⁰

249Lastly, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are mandated to reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of vulnerability exploitation and malicious code execution through ‘regular software updates and patch management’.⁷³¹ This can include ensuring that patches come from trusted sources, are tested before being applied and are applied within a reasonable time.⁷³²

2.1.4.3.3.2. Protection of unreleased model parameters

250A second component of the security requirements under Article 55(1)(d) – as reflected in Appendix 4.2 of the Safety and Security Chapter of the GPAI Code of Practice – concerns security mitigations that aim to protect unreleased model parameters. Securing unreleased model parameters is of high importance, since access to them by unauthorised and/or malicious actors may result in the model being deployed without the necessary systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessments and mitigations, thereby making the materialisation of systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. significantly more likely. The obligation to secure the model’s unreleased parameters thus contributes to ensuring the effectiveness of the measures adopted pursuant to Article 55(1)(a) and (b).⁷³³ That is because, even if the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s mitigations under those provisions are robust, once an attacker gains access to the models weights they can misuse them to operate the model without any restrictions or monitoring⁷³⁴ – thereby giving rise to systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Materials such as the RAND Securing AI Model Weights report (hereinafter “RAND report”) will be particularly useful for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. with regard to Appendix 4.2, not least because the Code of Practice explicitly lists the report as an example of ‘relevant guidance’.⁷³⁵

251First, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to achieve ‘accountability over all copies of stored model parameters across all devices and locations’ through ‘a secure internal registry of all devices and locations where model parameters are stored’.⁷³⁶ That makes sense because the securing of (copies of) model parameters will largely be ineffective if it is not clear where and how many copies of the model parameters exist in the first place.⁷³⁷ After all, a single unprotected copy is sufficient to circumvent the security measures adopted by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁷³⁸

252Further, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to prevent unauthorised copying of model parameters to unmanaged devices through ‘access management on all devices storing model parameters, with alerts in case of copying to unmanaged devices’.⁷³⁹ This measure is closely linked to the previously mentioned requirement to ensure accountability over all copies of the model parameters. Effective access management requires providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to ensure that all devices storing the model parameters are known. External guidance may offer examples of how such access controls can be implemented. For example, the RAND report recommends – in its Security Level (“SL”) 3⁷⁴⁰ – which has been referenced by earlier preparatory drafts of the GPAI Code of Practice⁷⁴¹ – storing weights in a small number of centrally managed locations so that employees and researchers ‘cannot simply make an additional copy’.⁷⁴² On this level, the report further suggests to protect all ‘sensitive interactions (including access to the weights themselves rather than using them for inference, and any editing of the code of the weights interface system)’.⁷⁴³ According to the RAND report, this can be facilitated, inter alia, by restricting the ability to make copies of the weights to 20 people, not granting third-party services access to the weights, not granting anyone persistent access, requiring multi-party authorisation and requiring security review for sensitive interactions.⁷⁴⁴ 

253Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are mandated to prevent unauthorised access to model parameters during transport and at rest through ‘ensuring model parameters are always encrypted during transportation and storage as appropriate, including encryption with at least 256-bit security and with encryption keys stored securely on a Trusted Platform Module (TPM)’.⁷⁴⁵  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. therefore need to at least make sure that they never use public or unencrypted channels for (plaintext) weight transport and that they secure their parameters at rest accordingly.⁷⁴⁶ 

254 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. should also aim to prevent unauthorised access to model parameters during temporary storage through ‘ensuring model parameters are only decrypted for legitimate use to non-persistent memory’.⁷⁴⁷ Model parameters need to be decrypted, generally said, when the parameters are needed for use – for example during training or fine-tuning – or when the model is being evaluated.⁷⁴⁸ What constitutes legitimate use can be informed by all other measures prescribed in the Code of Practice as well as other applicable laws.

255Closely related to that, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to prevent unauthorised access to model parameters during use through ‘implementing confidential computing as appropriate, using hardware-based, and attested trusted execution environments’.⁷⁴⁹ A trusted execution environment (“TEE”) ‘provides an isolated environment […] that safeguards processed data by encrypting the incoming and outgoing data’⁷⁵⁰ and ‘protects the data and computation against any potentially malicious entity residing in the system’.⁷⁵¹ This measure seems to be in line with SL4 in the RAND report,⁷⁵² which further concretises and suggests to ensure that the TEE includes protection against physical attacks, model weights are only encrypted by a key that is generated and stored in the TEE, and the TEE will only run pre-specified and audited signed code.⁷⁵³

256Lastly, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to prevent unauthorised physical access to systems that host model parameters through ‘restricting physical access to data centres and other sensitive working environments to required personnel only, along with regular inspections of such sites for unauthorised personnel or devices’.⁷⁵⁴ As physical access to systems that host model parameters will often be equivalent to access to parameters on the system,⁷⁵⁵  providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will have to operate with the same caution for physical access to systems storing parameters as they operate for the parameters themselves. Additionally, even if robust encryption is in place, physical access to systems can be a first step in enabling further attacks.⁷⁵⁶ Similarly, in SL3, for example, the RAND report suggests that the physical security entail that data centres are guarded and locked at all times and that premises are swept for intruders frequently and for unauthorised devices routinely.⁷⁵⁷

2.1.4.3.3.3. Hardening interface access to unreleased model parameters

257A third component of security requirements under Article 55(1)(d) – as reflected in Appendix 4.3 – deals with hardening interface access to unreleased model parameters while in use. These measures aim to protect the model’s parameters in use, because at that time they are specifically vulnerable – especially to be illegitimately copied – since the parameters are decrypted then.⁷⁵⁸ As already stated above, it is essential to ensure that there are no less-secure copies of the parameters in existence, as this would risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. undermining the whole systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation process pursuant to Article 55(1)(a) and (b).

258 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to prevent unnecessary interface access to models through ‘explicitly authorising only required software and persons for access to model parameters, enforced through multi-factor authentication mechanisms, and checked on a regular basis of at least every six months’.⁷⁵⁹ This measure is based on the observation that, in many organisations, a significant number of individuals have access to models.⁷⁶⁰ By restricting both the range of authorised software and the number of authorised users, the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of illegitimate copying is correspondingly reduced.⁷⁶¹

259Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of vulnerability exploitation or data leakage through ‘thorough review of any software interfaces with access to model parameters by a security team to identify vulnerabilities or data leakage, and/or automated security reviews of any software interface code at least to the same standard as the highest level of automated security review used for other sensitive code’.⁷⁶²

260Further, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of model parameter exfiltration through ‘hardening interfaces with access to model parameters, using methods such as output rate limiting’.⁷⁶³ Output rate limiting is an effective means of defending against parameter exfiltration because it ensures that the exfiltration of a significant portion of the weights would take too long to be practical.⁷⁶⁴ On SL4, the RAND report even suggests hardware-enforced limits on output rates.⁷⁶⁵

261Lastly, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of insider threats or compromised accounts through ‘limiting the number of people who have non-hardened interface-access to model parameters.’⁷⁶⁶ As noted above, it is essential to limit the number of people who have access to the model parameters through non-hardened interfaces to the necessary level.

2.1.4.3.3.4. Insider threats

262Another component of security requirements under Article 55(1)(d) – as reflected in the in the Safety and Security Chapter of the GPAI Code of Practice’s Appendix 4.4 – addresses protection against insider threats ‘including in the form of (self-)exfiltration or sabotage carried out by models’. GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. constitute highly attractive targets, making insider threats a realistic and significant risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. vector.⁷⁶⁷ Security that only faces external threats is therefore not sufficient for most, if not all providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might find guidance with regard to the following measures in the CISA’s Insider Threat Mitigation Guide⁷⁶⁸ and the National Insider Threat Task Force’s Insider Program Maturity Framework⁷⁶⁹, both of which are referenced in the RAND report.⁷⁷⁰

263First, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to protect the model’s parameters from insider threats attempting to gain work-related access through ‘background checks on employees and contractors that have or might reasonably obtain read or write access to unreleased model parameters or systems that manage the access to such parameters’.⁷⁷¹ The RAND report suggests having employees with parameter access to go ‘through extensive screening every six months’.⁷⁷² Further guidance on which indicators can be relevant in background checks more generally can be found in the CISA guide.⁷⁷³

264Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to raise awareness of the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of insider threats through ‘the provision of training on recognising and reporting insider threats’.⁷⁷⁴ Oftentimes, insider threats will only be detectable through the cooperation of employees who interact with their colleagues daily.⁷⁷⁵ A study found that in nearly 40% of employee data-exfiltration cases suspicious behaviour had been observed in advance by co-workers.⁷⁷⁶ This suggests that employee awareness forms a central component of any effective insider-threat mitigation strategy.⁷⁷⁷ The RAND report therefore recommends providing employees with guidance on what constitutes suspicious behaviour and on the appropriate reporting and response mechanisms.⁷⁷⁸ The CISA guide likewise offers practical orientation on the effective design and implementation of employee training and awareness programmes.⁷⁷⁹

265Further, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of model self-exfiltration through ‘sandboxes around models, such as virtual machines and code execution isolation’.⁷⁸⁰  Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must therefore not only have measures against internal human threats in place, but also against those threats arising from the model execution environment itself becoming a factor in parameter or data exfiltration.

266Lastly, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to reduce the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of sabotage to model training and use ‘through checking training data Article 3(29) AI Act: ‘training data’ means data used for training an AI system through fitting its learnable parameters. for indications of tampering’. Research indicates that data-poisoning attacks could be more practicable than previously assumed – finding that injecting as few as 250 malicious documents into pre-training data can suffice to introduce vulnerabilities to backdoor attacks across models of varying sizes.⁷⁸¹

2.1.4.3.3.5. Security assurance

267Under another component of the security requirement in Article 55(1)(d) – as reflected in Appendix 4.5 – providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will ‘obtain assurance that their security mitigations meet the Security Goal by implementing additional security mitigations’.⁷⁸²

268First, if the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s internal expertise is inadequate, they must achieve external validation of their security mitigation effectiveness through ‘regular independent external security reviews as appropriate to mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.⁷⁸³ On SL2, for example, the RAND report requires review and penetration testing by an ‘accredited third-party organization’.⁷⁸⁴ On SL3, it is suggested that the security team should perform continuous penetration testing, laying a focus on interfaces to the weights; penetration testing of physical access and ‘[a]dvanced red-teaming’.⁷⁸⁵ This entails having a highly capable external team⁷⁸⁶ which receives ‘significant funding’ and is given access to the system design and code so they can perform whitebox red-teaming.⁷⁸⁷ Additionally, those elite external teams should be given employee credentials to be able to test insider threats⁷⁸⁸ as well as expanded access in general.⁷⁸⁹

269 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’ security assurance obligations further encompass the validation of their network and physical access management as well as their security gap identification through ‘frequent red-teaming as appropriate to mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.⁷⁹⁰ Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are required to validate their network software integrity through ‘competitive bug bounty programs to encourage public participation in security testing of public-facing endpoints as appropriate to mitigate systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ’.⁷⁹¹

270 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will also have to validate their insider-threat security mitigations through ‘periodic personnel integrity testing’.⁷⁹² This resembles the RAND report’s SL4, suggesting occasional employee integrity testing,⁷⁹³ at the same time noting, however, that ‘the predictive reliability of different integrity testing approaches is unclear’.⁷⁹⁴ Moreover, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. facilitate the reporting of security issues through ‘secure communication channels for third parties to report security issues’.⁷⁹⁵ To detect suspicious or malicious activity, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will ‘install Endpoint Detection and Response (“EDR”) and/or Intrusion Detection System (IDS) tools on all networks and devices’.⁷⁹⁶

271Lastly, to be able to respond timely and effectively to malicious activity, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. will need to make ‘use of a security team to monitor for EDR alerts and conduct security incident handling, response, and recovery for security breaches in a timely and effective manner’.⁷⁹⁷

272In summary, the GPAI Code of Practice reflects a notably expansive understanding of cybersecurity within the meaning of Article 55(1)(d). Although its primary focus lies on the protection of model parameters, the concept of cybersecurity in Article 55(1)(d) in general must be understood in structurally broader terms – encompassing, for example, insider-threat mitigation, security assurance and general security mitigations. This approach is structurally consistent with the objectives of Article 55 as a whole: even minor weaknesses in the implemented cybersecurity mitigations as well as a narrow understanding of the concept of cybersecurity may ultimately render the measures taken pursuant to Article 55(1)(a) and (b) ineffective.

2.1.4.3.4. Synthesis

273Overall, it can be observed that Article 55(1)(d)’s scope, as clarified by the GPAI Code of Practice, is considerably broader than that of Article 15 AI Act. In particular, the Code’s understanding explicitly encompasses insider threats and addresses them in detail. Moreover, the Code is not limited to system vulnerabilities. Rather, it covers any potential weakness that could – even only multiple steps later – ultimately jeopardise the model’s systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigations. This broader concept is reflected, for example, in the Code’s attention to physical infrastructure and its general security mitigations, including, for example, policies on removable media. Finally, the Code is not confined to attempts aimed at altering the use, output or performance of systems, thereby extending beyond the narrower focus of Article 15.

274In sum, the broader conceptual approach adopted in the Code of Practice appears aligned with the broad legal definition set out in the CSA. This alignment is coherent: since GPAI models (with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. ) ‘may form the basis for a range of downstream systems’,⁷⁹⁸ it is particularly important that robust security standards apply already at the model level. Accordingly, the narrower definition underlying Article 15 should not be transposed to Article 55(1)(d).

2.1.4.4. Objective scope of protection: model and physical infrastructure

275The cybersecurity protection obligation applies to both ‘the general-purpose AI model Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. and the physical infrastructure of the model’.⁷⁹⁹ 

276Consistent with the broad understanding of cybersecurity mentioned above, the notion of a ‘model’ must likewise be interpreted broadly in this context.⁸⁰⁰ Rather than pursuing a formal definition of the term, the measures set out in the Code of Practice provide a useful indication of its objective scope. In light of the provision’s purpose – namely, ensuring that the measures adopted under Article 55(1)(a) and (b) remain effective and are not undermined – we can draw a few conclusions.

277First, in any case, the scope encompasses the unreleased parameters of the model, its algorithms, and all copies thereof.⁸⁰¹ This is further supported by Recital 115, which expressly refers to the securing of model weights (and algorithms). Illegitimate copies of parameters may enable the circumvention of systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigation measures and allow the model to be deployed without the safeguards required under Article 55(1)(a) and (b). Arguably, the notion of a model must likewise include any software interfaces that provide access to the model’s parameters. Effective protection of the model as such necessarily entails securing interfaces through which access to the model can be obtained.⁸⁰² Finally, the scope also extends to data used to train the model. Recital 115 refers to ‘securing […] data sets’ as one way of facilitating cybersecurity. Moreover, the GPAI Code of Practice also addresses the protection of training data Article 3(29) AI Act: ‘training data’ means data used for training an AI system through fitting its learnable parameters. by requiring signatories to mitigate the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of sabotage during model training and use, for example by ‘checking training data Article 3(29) AI Act: ‘training data’ means data used for training an AI system through fitting its learnable parameters. for indications of tampering’.⁸⁰³ This interpretation is also teleologically sound: tampered data might equally undermine the effectiveness of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigation measures undertaken pursuant to Article 55(1)(a) and (b).

278The AI Act also does not define exactly what is meant by the ‘physical infrastructure’ of a GPAI model with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . A first point of guidance for interpretation is Recital 115, which explicitly mentions ‘physical access controls’ as possible measures to ensure an adequate level of cybersecurity and refers to ‘servers’ as protected physical assets. Further guidance can additionally be drawn from the Code of Practice. It emphasises the prevention of unauthorised access to ‘systems hosting model parameters’, which, in its understanding, include ‘data centres and other sensitive working environments’.⁸⁰⁴ Beyond this, however, the contours of the model’s physical infrastructure remain unclarified. It is therefore necessary to revert to the purpose of Article 55(1)(d), which seeks to ensure effective systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. assessment and mitigation. On that basis, the concept of the model’s physical infrastructure should encompass all physical assets whose compromise or infiltration could ultimately undermine the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. measures in place. This would include, for example, portable devices, irrespective of whether model parameters are stored on them. As long as such devices could, if infiltrated, constitute a first step towards, for example, parameter exfiltration, and thereby threaten the effectiveness of the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigations in place, such devices should be encompassed by the obligation under 55(1)(d).

279On the other hand, an interpretation according to which all hardware and facilities that in any way support the operation of the model fall within the scope of Article 55(1)(d) would go too far. As noted above, it makes sense to interpret the notion of the model’s physical infrastructure as covering those elements whose insufficient protection can lead to an increase in the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. emanating from the model. For instance, the release of model weights could lead to a loss of control over a powerful model and may enable, for example, cyber-offence risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. .⁸⁰⁵ An inadequate protection of supporting facilities such as ancillary facilities unconnected to parameter storage, power systems or cooling infrastructure, on the other hand, may certainly be relevant for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. more generally.⁸⁰⁶ However, it seems inappropriate to argue that such protection duties fall under Article 55(1)(d), as these risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. do not primarily amplify or extend the systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. by undermining the systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. mitigation of the respective model itself, at least as long as compromise or infiltration of such physical assets can ultimately not lead to increased systemic risks Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. stemming from the model – for example because power systems are completely separated from facilities in which model parameters are stored and the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. additionally ensures that power outages cannot undermine the parameters’ security.

2.1.4.5. The providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’ obligation to ‘ensure’ an adequate level of cybersecurity

280It could be argued that the cybersecurity obligation can only apply to physical infrastructure that is under direct control of the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. .⁸⁰⁷ However, this can be countered by the fact that Article 55(1)(d) states that the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must ‘ensure’ an adequate level of cybersecurity. This can be understood to mean that the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must guarantee, for example through contractual agreements including control rights, that sensitive working infrastructure not directly under its direct, factual control also meet an adequate level of security. Additionally, it can be argued that it would be contradictory to Article 55’s goals if the obligations of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. were reduced as their control diminishes. Such an approach would create incentives for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to outsource as much as possible to third-party infrastructure (which they can not control), thereby undermining AI safety and reliability.

2.1.4.6. Adequate level of cybersecurity

281 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must ensure an ‘adequate’ level of cybersecurity.⁸⁰⁸ In Recital 115, the AI Act appears to further clarify how the adequacy of cybersecurity is to be assessed. It states that protection should be ‘appropriate to the relevant circumstances and the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. involved’.

282The GPAI Code of Practice provides some further, valuable input in this regard. It states that signatories will ‘define a goal that specifies the threat actors that their security mitigations are intended to protect against (“Security Goal”), including non-state external threats, insider threats, and other expected threat actors, taking into account at least the current and expected capabilities of their models’.⁸⁰⁹ Additionally, the Code clarifies that the ‘implementation of the required security mitigations may be staged appropriately in line with the increase in model capabilities along the entire model lifecycle’.⁸¹⁰ This Security Goal plays an important role in the Safety and Security Model Reports the signatories commit to report to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. .⁸¹¹ That is because, under Measure 7.3(3) of the Safety and Security Chapter, signatories commit to describing their Security Goal, all security mitigations they implemented, and how those measures meet the Security Goal.

283Importantly, the latter includes ‘the extent to which they align with relevant international standards or other relevant guidance (such as the RAND Securing AI Model Weights report)’.⁸¹² This raises the broader question of how the general requirement of an ‘adequate level’ of cybersecurity is to be understood – and in particular what role the state of the art plays in this context. In this regard, the Code of Practice appears instructive: whilst Article 55 refers to the state of the art in Article 55(1)(a), it does not define it.⁸¹³ The Code of Practice, by contrast, defines it in its Glossary as ‘the forefront of relevant research, governance, and technology that goes beyond best practice’.⁸¹⁴ This forefront understanding is further underlined in Recital (a) of the Code, according to which signatories must generally adopt ‘at least’ the state of the art in order to implement appropriate measures.⁸¹⁵ Signatories are furthermore encouraged to advance the state of the art.⁸¹⁶ This seems to suggest that the requirement of an ‘adequate level’ of cybersecurity under Article 55 is to be understood in light of this forefront standard: adequacy is thus not satisfied merely by complying with established best practices but requires an orientation towards the forefront of current research. Against this backdrop,⁸¹⁷ the aforementioned RAND Securing AI Model Weights report may nevertheless serve as a good starting point for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to draw upon – not least because it extends to security levels that are not yet fully achievable and that aim to thwart operations by actors that have ‘experience and expertise years ahead of the (public) state of the art’,⁸¹⁸ thereby reflecting a standard upon which future forefront research may build.

284As noted above and since the GPAI Code of Practice explicitly addresses the RAND Securing AI Model Weights report as relevant guidance, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are likely best advised to rely on it as a primary point of orientation. The RAND report identifies 38 distinct attack vectors, distinguishes between a variety of potential threat actors and their respective capabilities, and, on that basis, proposes five distinct security levels accompanied by preliminary benchmarks.⁸¹⁹ The threat actor modelling envisaged under the Code’s Security Goal can, in practice, orientate on and align with the Security Levels set out in the RAND report. These levels are structured according to the type and sophistication of attackers against whom a model must be secured. This approach also seems consistent with Recital 115, which refers, more generally, to the ‘ risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. involved’: both the threat landscape and the risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. to the model are likely to scale with increases in the model’s capabilities. The more capable a model, the greater the likelihood that highly sophisticated actors will seek to compromise it. The SLs in the RAND report range from SL1 (‘A system that can likely thwart amateur attempts’) to SL5 (‘A system that could plausibly be claimed to thwart most top-priority operations by the top cyber capable institutions’).⁸²⁰ 

285While the third draft of the GPAI Code of Practice referred more explicitly to the RAND report in the context of the security mitigations under (then) Commitment II.7,⁸²¹ the final adopted version of the Code no longer contains an express reference to the RAND report under Measure 6 or Annex 4. Nevertheless, the final Code still, as mentioned above,⁸²² continues to emphasise the relevance of the RAND report, such that the guidance contained in the third draft may still serve as a useful interpretive aid. Under the third draft, signatories were expected to ‘meet at least RAND SL3 or equivalent’. This seems to be a plausible first point of orientation for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. under the final GPAI Code of Practice and under Article 55(1)(d) as well. This is further supported by the fact that many of the security mitigations outlined above appear to be modelled, at least in part, on the measures associated with the RAND report’s SL3 benchmark. At the same time, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. must ensure that their threat modelling under the Security Goal adequately considers whether reasonably foreseeable threats may also stem from SL4 or even SL5 adversaries.⁸²³ In this regard, it should be noted that the growing availability of open-weight GPAI models has the potential to reshape the cybersecurity threat landscape, as both the capabilities and the actor profiles – such as those described in the RAND report – may evolve and escalate.⁸²⁴

286Other ‘relevant international standards’ and ‘relevant guidance’ providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may build upon to ensure an adequate level in the aforementioned sense could especially include ISO/IEC, NIST or SOC publications.⁸²⁵ Expressly mentioned by the third draft of the Code were ISO/IEC 27001, NIST 800-53 and SOC 2.⁸²⁶ With regard to AI-specific risks Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. , orientation might be drawn from ISO/IEC TR 27563:2023⁸²⁷ or ISO/IEC DIS 27090⁸²⁸. Additionally, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might find guidance in publications by ENISA, given the agency’s mandate under the CSA; for example, ENISA’s yearly threat landscape report may offer guidance for modelling the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ’s Security Goal.

2.1.4.7. Exemption for less capable, publicly available and deleted models

287Importantly, the Safety and Security Chapter of the GPAI Code of Practice exempts models from Commitment 6 – the security mitigations – in three distinct scenarios. First, models are exempt from the commitment where ‘the model’s capabilities are inferior to the capabilities of at least one model for which the parameters are publicly available for download’ (inferior-capabilities exemption).⁸²⁹ Second, the Code only requires signatories to implement security mitigations for a model ‘until its parameters are made publicly available for download’ (open access exemption).⁸³⁰ Third, in the same sense, models are no longer subject to the exemption as soon as their parameters are ‘securely deleted’ (deletion exemption).⁸³¹

288The rationale underlying these exemptions in the Code appears to be that the primary objective of the security mitigations – namely, to protect model weights and thereby prevent the deployment or misuse of a highly capable model in the wrong hands – may no longer be necessary, or at least proportionate, where potential malicious actors can in any event readily download the parameters of an even more capable model, where the model’s parameters are made available for download, and – obviously – where the model’s parameters have securely been deleted.

289Two aspects of the inferior-capabilities exemption appear challenging. First, the Code does not define when a model’s capabilities are to be considered inferior to those of another model. Second, it remains unclear who is to make this determination and whether, and to what extent, such an assessment is subject to verification. With regard to model capabilities more generally, signatories will likely rely on the aspects listed in Appendix 1.3.1 as relevant indicators. However, it remains uncertain to what extent a reliable comparison between the signatory’s own model and a potentially more capable external model is feasible in practice – the risk Article 3(2) AI Act: ‘risk’ means the combination of the probability of an occurrence of harm and the severity of that harm. of misclassification residing with the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. . Second, it remains unclear how this exemption in the GPAI Code of Practice affects non-signatories. One could argue that, since the Code does not apply to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who have not signed it, they cannot invoke the exemption in the first place. In reviewing whether a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. can demonstrate ‘alternative adequate means of compliance’ under Article 55(2), the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. could, in theory, take the position that it is not bound by an exemption formulated in the Code. On that view, the exemption might simultaneously function as an incentive to adhere to the Code. On the other hand, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. could argue that the same rationale – disproportionality to impose security mitigations in cases where a more capable model is freely available – must also inform the interpretation of ‘adequate’ in Article 55(1)(d), thereby leading to the same outcome as if an exemption applied. A position taken by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. according to which they are not bound to an exemption in the Code would likely also violate the legitimate expectations of providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. (arguably, including non-signatories),⁸³² since the Commission has expressly stated that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ‘can demonstrate compliance with the obligations in Articles 53(1) and 55(1) AI Act by adhering to a code of practice that is assessed as adequate by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the Board.’⁸³³

290The two other exemptions – the open access exemption and the deletion exemption – appear less challenging yet are not entirely without difficulties. The former could be seen as creating an incentive for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to proactively release their model weights in order to escape their obligations under Article 55(1)(d). The deletion exemption raises a distinct concern: the GPAI Code of Practice does not define when parameters are to be considered ‘securely deleted’, nor does it specify the technical standard against which such deletion is to be assessed. While general guidance on secure deletion exists – most notably NIST SP 800-88,⁸³⁴ which addresses methods such as cryptographic erasure and data overwriting – this standard was not designed with AI model weights in mind. The question of how the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. is to verify compliance with the deletion exemption remains similarly unresolved.

2.1.4.8. Temporal scope

291Although not apparent from Article 55(1)(d) itself, it follows from Recital 115 that the cybersecurity obligation applies ‘along the entire model lifecycle’ – but only ‘if appropriate’. It remains unclear why this important clarification is included only in the (non-binding) recitals and not in the legislative text. For signatories of the Code of Practice, this additionally follows from Commitment 6 of the Safety and Security Chapter, according to which they ‘commit to implementing an adequate level of cybersecurity protection for their models and their physical infrastructure along the entire model lifecycle’ (emphasis added).⁸³⁵

292The Commission tends to favour a broad interpretation of the term. In its GPAI Guidelines, it acknowledges that it is difficult ‘to clearly delineate a model and its lifecycle’ because of the ‘iterative and interlinked process through which a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. may develop a “model”’.⁸³⁶ This is why the Commission understands ‘the notion of a “Model”, and consequently its “lifecycle” in a broad sense’.⁸³⁷ According to the GPAI Guidelines, the lifecycle of a model begins ‘at the start of the large pre-training run’.⁸³⁸ Importantly, the Commission states that any ‘subsequent development of the model downstream of this large pre-training run performed by the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. or on behalf of the provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , whether before or after the model has been placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. , forms part of the same model’s lifecycle rather than giving rise to new models’.⁸³⁹

293The GPAI Code of Practice does not explicitly address how exactly it defines the lifecycle of a model. Recital (a), the ‘Principle of Appropriate Lifestyle Management’, does clarify that the model lifecycle includes the ‘development that occurs before and after a model has been placed on the market Article 3(9) AI Act: ‘placing on the market’ means the first making available of an AI system or a general-purpose AI model on the Union market. ’. Much like the Commission’s GPAI Guidelines, the Code therefore appears to be based on a broad understanding of the model lifecycle.

294A difficulty that arises not only in this context is that, according to Recital 115, the obligation to ensure an adequate level of cybersecurity applies ‘along the entire lifecycle’ of the model, whereas Article 2(8) provides an exemption from the scope of the AI Act for research, development and testing activities. This tension is discussed in more detail elsewhere.⁸⁴⁰

2.2. Article 55(2): Compliance pathways

295Article 55(2) details some of the ways in which providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models presenting systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can comply with their obligations under Article 55(1). More specifically, it details three distinct compliance pathways: harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. , codes of practice and alternative adequate means. In choosing a compliance pathway, providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. are, to some extent, limited to those pathways that have been formally adopted by the relevant authorities.

296Regarding the former, it is clear that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models presenting systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. cannot rely on a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. in the absence of such a standard. Article 55(2) expressly acknowledges this in its first sentence, by directing providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. to instead rely on a code of practice ‘until’ a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. is published. However, the availability of this pathway was itself contingent on the prior creation of such a code of practice. While a code of practice was eventually created,⁸⁴¹ Article 56(9) contemplates the possibility that no such code would exist or that it would be considered inadequate and provides that, in that scenario, the Commission may provide common rules by way of an implementing act.⁸⁴²

297Depending on the availability and adoption of these compliance pathways at a given moment, it is important to assess who may, can or must rely on them and with what legal effects. We will briefly touch on this question of provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. discretion throughout the remainder of this assessment, referring the reader elsewhere for a more extensive discussion.⁸⁴³ On a general level, it is clear that the notion of legitimate expectations plays a key role regarding the question of whether providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. can rely on these instruments.⁸⁴⁴ More specifically, the fact that the Commission has assessed a given code of practice as adequate may preclude the Commission from arguing that a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who did not sign the relevant code but nevertheless adheres to it regarding some AI Act obligation has violated that obligation.⁸⁴⁵ Whether providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. have to rely on these instruments, or whether the avenue of ‘alternative adequate means’ remains an option is, in general, more difficult to assess,⁸⁴⁶ as will become clear below.

2.2.1. Harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012.

298The AI Act positions harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. as the final and principal compliance pathway for providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. . Article 3(27) clarifies that the notion of ‘ harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. ’ is to be understood in the sense of Article 2(1)(c) of Regulation (EU) No 1025/2012,⁸⁴⁷ which defines a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. as ‘a European standard adopted on the basis of a request made by the Commission for the application of Union harmonisation legislation’.⁸⁴⁸

299In the European Union, such standards are developed by three bodies: the European Committee for Standardisation (“CEN”), the European Committee for Electrotechnical Standardisation (“Cenelec”) and the European Telecommunications Standards Institute (“ETSI”).⁸⁴⁹ These standards are normally expected to reflect the state of the art.⁸⁵⁰ Given the fast-paced evolutions in the field of AI and its evaluations,⁸⁵¹ and the fact that it typically takes standardisation bodies time to develop a standard (amongst others reasons because of the importance to consultant relevant stakeholders, as discussed below),⁸⁵² it is unclear how this reference to the state of the art could reflect ‘the forefront of relevant research, governance, and technology that goes beyond best practice’,⁸⁵³ the definition that is applied to the state-of-the-art condition in Article 55(1)(a).⁸⁵⁴ Instead, this reference likely⁸⁵⁵ refers, more broadly, to ‘generally acknowledged state of the art’, the definition as it is typically deployed in the context of standardisation, reflecting ‘what is currently and generally accepted as good practice.’⁸⁵⁶

300The time-consuming process required to get standards ‘right’ implies that not all of the relevant obligations in Article 55(1) can reasonably be ‘standardised’.⁸⁵⁷ A notable example is thus 55(1)(a)’s reference to the state of the art in the sense of the forefront of relevant research, governance, and technology that goes beyond best practice.⁸⁵⁸ This inherently more limited role for some of the Article 55 obligations means that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. will not be able to fully rely on such standards. It is likely that this will, in practice, lead to a continued role for codes of practice alongside some standards if and when those are developed, nuancing Article 55(2)’s textual implication that the role of codes of practice is limited to serve as a placeholder – a ‘temporary tool’⁸⁵⁹ – until if and when such a standard is published. Nevertheless, to the extent that a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. does cover an aspect that is equally covered by a code of practice, the standard clearly prevails.⁸⁶⁰

301The relevant standards are to be created after a standardisation request by the European Commission⁸⁶¹ and should ideally be based on a ‘balanced representation of interests involving all relevant stakeholders in the development of standards, in particular SMEs, consumer organisations and environmental and social stakeholders’.⁸⁶² This process thus entails similar stakeholders as for the development of codes of practice, as Article 56(3) indicates that ‘[t]he AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. may invite all providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of general-purpose AI models Article 3(63) AI Act: ‘general-purpose AI model’ means an AI model, including where such an AI model is trained with a large amount of data using self-supervision at scale, that displays significant generality and is capable of competently performing a wide range of distinct tasks regardless of the way the model is placed on the market and that can be integrated into a variety of downstream systems or applications, except AI models that are used for research, development or prototyping activities before they are placed on the market. , as well as relevant national competent authorities Article 3(48) AI Act: ‘national competent authority’ means a notifying authority or a market surveillance authority; as regards AI systems put into service or used by Union institutions, agencies, offices and bodies, references to national competent authorities or market surveillance authorities in this Regulation shall be construed as references to the European Data Protection Supervisor. , to participate in the drawing up of codes of practice. Civil society organisations, industry, academia and other relevant stakeholders, such as downstream providers Article 3(68) AI Act: ‘downstream provider’ means a provider of an AI system, including a general-purpose AI system, which integrates an AI model, regardless of whether the AI model is provided by themselves and vertically integrated or provided by another entity based on contractual relations. and independent experts, may support the process’.⁸⁶³

302On their surface,⁸⁶⁴  harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. hold the key advantage over codes of practice that compliance with the former grants providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. a presumption of conformity with Article 55(1), whereas adherence to the latter does not.⁸⁶⁵ This presumption of conformity is likely rebuttable.⁸⁶⁶ This difference, however, is, at least in part, nuanced for codes of practice that were deemed adequate by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the Board,⁸⁶⁷ because of the Commission’s view that such codes can be used to ‘demonstrate compliance with the obligations in Articles 53(1) and 55(1) AI Act’.⁸⁶⁸

303Recital 117 indicates that the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. will assess the suitability of harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. , implying that the presumption of conformity only applies if and when the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. has done so.⁸⁶⁹ This is not reflected in the text of Articles 55 and 56 but, instead, likely⁸⁷⁰ reflects the Article 40(1) and Regulation 1025/2012 requirement that harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. undergo Commission (and standardisation body) assessment⁸⁷¹ before a reference to those standards is published in the Official Journal⁸⁷² and thus entail a presumption of conformity.

2.2.2. Codes of practice

304Absent a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. , Article 55(2) indicates that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models with systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. can rely on codes of practice to demonstrate compliance with Article 55 first paragraph’s obligations until a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. is published. Article 55(2)’s last sentence clarifies that this necessitates the code to have been ‘approved’ – by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the Board⁸⁷³ – to serve this compliance function.⁸⁷⁴

305It should be noted, as discussed earlier, that Article 55(2) does not extend the presumption of conformity that applies to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. adhering to a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. to apply to providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that rely on an approved code of practice.⁸⁷⁵ As such, there is no general presumption that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. that adhere to an approved code of practice are compliant with the AI Act.⁸⁷⁶ Nevertheless, this distinction appears to be largely theoretical,⁸⁷⁷ as the Commission Guidelines state that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ‘can demonstrate compliance with the obligations in Articles 53(1) and 55(1) AI Act by adhering to a code of practice that is assessed as adequate by the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and the Board’.⁸⁷⁸

306Lastly, although it is not covered in detail here,⁸⁷⁹ Article 56(6) also indicates that an approved code of practice can be given general validity by way of an implementing act.

307It is clear that approved codes of practice offer important guidance when it comes to the interpretation of the AI Act’s provisions, even if a provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. did not sign on to that code of practice.⁸⁸⁰ The Commission Guidelines indicate as much, by stating that the alternative adequate means adopted by such providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , discussed below, can be shown to result in AI Act compliance, ‘for instance by carrying out a gap analysis that compares the measures they have implemented with the measures set out by a code of practice that is assessed as adequate.’⁸⁸¹ The Commission Guidelines also indicate that the ‘Commission may take into account commitments implemented in line with a code of practice that is assessed as adequate as a mitigating factor when fixing the amount of fines, depending on the specific circumstances’.⁸⁸² 

2.2.3. Alternative adequate means

308 Providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. of GPAI models that present systemic risk Article 3(65) AI Act: ‘systemic risk’ means a risk that is specific to the high-impact capabilities of general-purpose AI models, having a significant impact on the Union market due to their reach, or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or the society as a whole, that can be propagated at scale across the value chain. are not per se required to adhere to an approved code of practice or harmonised standards Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. .⁸⁸³ While the situation is less clear for codes of practice that are given general validity by way of implementing act,⁸⁸⁴ Article 55(2) recognises that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. ‘who do not adhere to an approved code of practice or do not comply with a European harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. shall demonstrate alternative adequate means of compliance’.

309While the AI Act does not directly describe what ‘alternative adequate means’ entail, this notion clearly refers to a broad category of measures that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. might take to ensure and, perhaps more importantly, demonstrate⁸⁸⁵ their compliance with Article 55(1). The latter is also evident from the Commission’s position that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who do not adhere to an approved code of practice (or a harmonised standard Article 3(27) AI Act: ‘harmonised standard’ means a harmonised standard as defined in Article 2(1), point (c), of Regulation (EU) No 1025/2012. ) are expected to report to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. how the measures they have implemented ensure compliance with the AI Act,⁸⁸⁶ though the AI Act does not contain an explicit obligation in this sense.

310Despite the inherently broad nature of ‘alternative adequate means’, the Commission Guidelines do reflect the Commission’s expectation that these means would be similar to those found in an approved code of practice.⁸⁸⁷ More specifically, when reporting to the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. how providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. comply with the AI Act through alternative adequate means, the EU Commission expects them to explain how their measures ensure compliance with the Act’s obligations, listing, as an example, ‘by carrying out a gap analysis that compares the measures they have implemented with the measures set out by a code of practice that is assessed as adequate.’⁸⁸⁸

311The Commission Guidelines (thus) strongly imply that adherence to an approved code of practice – even when that code of practice was not signed on to by a specific provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. – constitutes the most straightforward pathway to compliance. In this sense, they indicate that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. who do not adhere to an approved code may expect more requests for information and access,⁸⁸⁹ as their choice not to rely on the approved code complicates the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ’s compliance assessment.⁸⁹⁰ Moreover, code of practice commitments that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. do adhere to are described as a mitigating factor in case of a potential fine for non-compliance, as discussed above.⁸⁹¹

2.3. Article 55(3): Confidentiality

312Article 55(3) requires that the Commission treat the information and documentation it obtains by virtue of Article 55 confidentially, in accordance with the obligations set out in Article 78. In this regard, Article 78(2) is particularly relevant. It requires the Commission to limit their requests to data that is ‘strictly necessary’ for the exercise of its powers under the AI Act. It also requires them to put in place adequate and effective cybersecurity measures to protect the security and confidentiality of the information and data they obtain, and to delete such data as soon as it is no longer required to assess compliance.

313Before looking at some of these requirements in more detail, it is interesting to note that a direct consequence of this confidentiality requirement is that the general public will not have access to the information that providers Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. share with the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. . As such, they will, for example, not have access to submitted information about serious incidents Article 3(49) AI Act: ‘serious incident’ means an incident or malfunctioning of an AI system that directly or indirectly leads to any of the following: (a) the death of a person, or serious harm to a person’s health; (b) a serious and irreversible disruption of the management or operation of critical infrastructure; (c) the infringement of obligations under Union law intended to protect fundamental rights; (d) serious harm to property or the environment. .⁸⁹²

2.3.1. Strict necessity

314First of all, Article 78(2) clarifies that the AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. and Commission shall only request ‘data’ that is ‘strictly necessary’ for the exercise of their powers under the AI Act.⁸⁹³ While some have argued that Article 91(1) contains a more specific applicable provision which would overrule Article 78(2) in the contexts of Articles 53 and 55,⁸⁹⁴ that consideration is offset by the latter provisions’ express reference to Article 78, strongly implying the applicability of the latter.⁸⁹⁵ Article 91(1) empowers the Commission with a broader mandate,⁸⁹⁶ allowing it to request all information ‘necessary’ for compliance assessments, whereas Article 78(2) only permits it to seek information that is ‘strictly necessary’ for the purposes of exercising the same compliance assessment powers and duties. An argument in favour of sustaining the superlative condition of ‘strictly necessary’ is the obvious sensitivity of the information the Commission is able to request under this power.⁸⁹⁷

315The requirement of necessity – and the present requirement of strict necessity, more specifically – imposes a balancing exercise between the Commission’s need for information to assess compliance with the AI Act, on the one hand, and various fundamental rights of the model provider Article 3(3) AI Act: ‘provider’ means a natural or legal person, public authority, agency or other body that develops an AI system or a general-purpose AI model or that has an AI system or a general-purpose AI model developed and places it on the market or puts the AI system into service under its own name or trademark, whether for payment or free of charge. , on the other hand. The latter include Article 17(2) of the Charter, which protects intellectual property (including trade secrets⁸⁹⁸), Article 7 of the Charter, which protects the right to privacy and which applies to legal persons as well,⁸⁹⁹ and Article 16 of the Charter’s protection of the freedom to conduct a business.⁹⁰⁰ These fundamental rights imply that any interference within the sphere of private activities of a party is to be proportionate and deliberate.⁹⁰¹

316The term ‘strictly necessary’ indicates that the proportionality test must be applied with a higher level of scrutiny than when the standard is merely ‘necessary’. Where ‘necessary’ already implies that the information requested is required to pursue the AI Act’s aims and that there is no less restrictive equally effective alternative available,⁹⁰² ‘strictly necessary’ goes beyond that and implies that the Commission and AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. enjoy a narrower margin of discretion. There should thus be some particular circumstance that supports and warrants the Commission and AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. ’s request – extending beyond the general idea that more information might help the Commission and AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. better exercise their mandate.⁹⁰³

317Resultantly, the phrase ‘strictly necessary’ implies an important procedural consequence. On this basis, the Commission and AI Office Article 3(47) AI Act: ‘AI Office’ means the Commission’s function of contributing to the implementation, monitoring and supervision of AI systems and general-purpose AI models, and AI governance, provided for in Commission Decision of 24 January 2024; references in this Regulation to the AI Office shall be construed as references to the Commission. can only make a reasoned request to exercise this option.⁹⁰⁴ That request should thus indicate the reasons why the relevant information is strictly necessary.⁹⁰⁵

2.3.2. Cybersecurity

318Article 78(2) requires ‘authorities’ – the Commission and the AI Offices in particular, in this context – to ‘put in place adequate and effective cybersecurity measures to protect the security and confidentiality of the information and data obtained’.⁹⁰⁶ The need for cybersecurity is a natural corollary of the need for confidentiality; the latter being a natural implication of the general principle of the protection of business secrets.⁹⁰⁷

319The AI Act’s cybersecurity requirement mimics similar requirements found in other European legislation, such as under Regulation (EU) 2018/1725⁹⁰⁸ which governs personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. processing by Union institutions, bodies, offices and agencies. In its Article 36, the latter imposes confidentiality and related requirements. Its Article 32 imposes security requirements that, inter alia, relate to the confidentiality and integrity of processing systems and services. It is also noteworthy to signal that the latter is applicable in the context of the AI Act, if personal data Article 3(50) AI Act: ‘personal data’ means personal data as defined in Article 4, point (1), of Regulation (EU) 2016/679. is involved.⁹⁰⁹ Other relevant instruments are Directive (EU) 2022/2555,⁹¹⁰ and, less directly, Regulation (EU, Euratom) 2023/2841,⁹¹¹ which impose cybersecurity requirements on public and private entities and on Union institutions, bodies, offices and agencies, respectively. 

320The comparison with these instruments – which, while useful, has clear limitations, for example due to the different contexts and objectives of some of them, such as Regulation 2018/1725, when compared to the AI Act⁹¹² – is particularly interesting as some of their more explicit requirements offer more context that can help interpret the similar obligation under the AI Act. Most notably, various of these instruments indicate that the cybersecurity measures at hand should live up to the state of the art⁹¹³ – which is to be understood as referring to the ‘generally acknowledged state of the art’.⁹¹⁴ In this respect, ISO 27001 is particularly interesting (as well as ISO 27002), though not directly applicable, as this sets out the relevant standards for information security management systems.⁹¹⁵