Whilst AI systems face many of the same security risks as traditional software, they also face a number of unique ones. Amongst these are the increased size and complexity of the AI attack surface given the need for additional hosting infrastructure, interconnected systems and software dependencies, for example. Novel attacks – which are generally grouped into poisoning, extraction or evasion – also need to be considered, alongside standard threats.
There’s a recognition that AI-related cyber security risks are currently underestimated, and that more needs to be done to help organisations to understand and assess those risks, mindful that security is key to building trust, and a lack of trust risks jeopardising AI adoption.
These concerns have prompted “a two-part intervention” by the UK government, which published its AI Cyber Security Code (‘Code’) on 31 January 2025 – a voluntary Code of Practice focused on securing AI systems throughout their lifecycle.
Given the government’s stated aim in its AI Opportunities Action Plan to position the UK as “an AI maker, not an AI taker”, it’s not at all surprising to learn that the Code is intended as the precursor to the creation of a global standard that sets baseline security requirements.
It’s an intervention in two parts because the Code is accompanied by a useful implementation guide that uses the following scenarios to provide examples of practical ways to meet those requirements:
- an organisation that uses a publicly available LLM via the APIs offered by the external provider to develop a chatbot for internal and customer use;
- a mid-size software company that selects an open-access classification model, which they train further with additional datasets to develop and host a fraud detection system;
- a tech company that develops a multimodal LLM capable of understanding and generating text, audio and images, and provides commercial API access to developers for diverse applications, such as virtual assistants and media generation;
- a small organisation that develops an LLM for specific use cases including for open access use for legal and contract negotiations, use by a law firm for legal research, and use by a rural development organisation for localised advice on crop management and pest control.
The Code itself consists of 13 Principles, broken down into 5 phases of the AI lifecycle, and applies to different stakeholders – principally Developers (i.e. providers, in EU AI Act lingo), and System Operators (i.e. deployers).
For ease of reference, here’s a table summarising these elements (NB corresponding stakeholders to whom each Principle applies are referred to as follows: Developers = ‘D’; System Operators = ‘SO’).
| Phase | No. | Description of Principle | Summary of Principle |
| DESIGN | 1 | Raise awareness of AI security threats and risks |
Include AI security in cyber security training, tailored to specific roles and responsibilities of staff. [D + SO] Ensure staff are aware of the latest AI-related threats and mitigations, with updates communicated through multiple channels. [D + SO] Train developers in secure coding and system design techniques specific to AI. [D + SO] |
| 2 | Design your AI system for security as well as functionality and performance |
Conduct thorough assessments of business requirements, along with AI security risk and mitigation strategies. [D + SO] Design AI systems to withstand adversarial AI attacks, unexpected inputs and AI system failure. [D + SO] Document and create an audit trail. [D] Involve data custodians to ensure appropriate usage of data, and encourage proactive risk reporting by employees. [D + SO] Risk assess system permissions and conduct due diligence on providers and external components. [D + SO] |
|
| 3 | Evaluate the threats and manage the risks to your AI system |
Analyse threats and manage security risks, ensuring threat modelling addresses AI-specific attacks (e.g. data poisoning, model inversion, and membership inference). [D + SO] Manage risks of superfluous functionalities (e.g. where only a single modality of a multimodal model is used). [D] Apply controls based on considerations, including cost of implementation and corporate risk tolerance. [SO] Communicate unresolved threats. [D + SO] Attain assurance from external entities responsible for AI security risks. [SO] Continuously monitor and review system infrastructure, recognising that a higher level of risk will remain in AI systems. [D + SO] |
|
| 4 | Enable human responsibility for AI systems |
Incorporate human oversight capabilities (with technical measures). [D + SO] Make it easy for humans to assess outputs (i.e. by ensuring outputs are explainable/interpretable). [D] Verify that security controls specified by the data custodian are built in. [D] Inform end-users of prohibited use cases. [D + SO] |
|
| DEVELOPMENT | 5 | Identify, track and protect your assets |
Maintain a comprehensive inventory of assets (including interdependencies/connectivity). [D + SO] Track and secure those assets. [D + SO] Ensure disaster recovery plans account for AI-specific attacks. [SO] Protect sensitive data against unauthorised access, and apply checks and sanitation to data and inputs. [D + SO] Put proportionate protection in place for confidential training data or model weights. [D] |
| 6 | Secure your infrastructure |
Evaluate access control frameworks and appropriately secure APIs, models, data, and training and processing pipelines. [D + SO] Apply controls that mitigate attacks via the API (e.g. limit model access rate). [D] Create dedicated environments for development and model-tuning, with technical controls to ensure separation and least privilege. [D] Put in place a vulnerability disclosure policy. [D + SO] Maintain and test plans for AI system incident management plan and recovery. [D + SO] Ensure contracts with cloud service operators support compliance with these requirements. [D + SO] |
|
| 7 | Secure your supply chain |
Follow secure software supply chain processes. [D + SO] Decisions to use models that are not well documented or secured must be justified and documented, and documentation shared with end users. [SO] Conduct risk assessments and ensure mitigating controls for such models. [D + SO] Re-run evaluations on released models. [D + SO] Communicate to end users an intention to update models. [SO] |
|
| 8 | Document your data, models and prompts |
Document and maintain clear audit trails of system design and maintenance plans, and make documentation available downstream. [D] Ensure documentation includes sources of training data, intended scope and limitations, guardrails, retention time, suggested review frequency and potential failure modes. Release cryptographic hashes for model components. [D] Given the risk of poisoning, document how publicly sourced data for training were obtained and used in the model (e.g. URL of the scraped page with date/time). [D] Log changes to system prompts or other configurations and make the audit log accessible. [D] |
|
| 9 | Conduct appropriate testing and evaluation |
Ensure all models, applications and systems undergo security testing before release/deployment, using independent security testers with relevant technical skills. [D + SO] Share findings with operators. [D] Evaluate model outputs to prevent reverse engineering and unintended influence. [D] |
|
| DEPLOYMENT | 10 | Communication and processes associated with End-users and Affected Entities |
Inform end-users how their data will be used. [SO] Provide them with accessible guidance on appropriate use of AI systems (including limitations and failure modes) [SO], with developers to provide all necessary information to help. [D] Support during and after cyber security incidents, with the process documents agreed in contracts with end-users. [D + SO] |
| MAINTENANCE | 11 | Maintain regular security updates, patches and mitigations |
Provide security updates and patches, with notifications. [D + SO] Treat major updates as new versions requiring security testing and evaluation. [D] Support evaluation and response to model changes (e.g. preview access via beta-testing and versioned APIs). [D] |
| 12 | Monitor your system's behaviour |
Log system and user actions to support security compliance, incident investigations, and vulnerability remediation. [SO] Analyse logs for anomalies, security breaches, or unexpected behaviour over time (e.g. data drift or data poisoning). [SO] Monitor internal states of AI systems. [D + SO] Monitor performance over time to detect changes in behaviour. [D + SO] |
|
| END OF LIFE | 13 | Ensure proper data and model disposal | Involve data custodians and securely dispose of training data and/or models when transferring/sharing ownership or decommissioning them. [D + SO] |
Analysis
Following the Paris AI Summit, the French National Cyber Security Authority published a paper called Building trust in AI through a cyber risk-based approach, which was co-signed by partners including the UK’s National Cyber Security Centre.
This paper provides an overview of key risks and attack scenarios, along with a user-friendly Appendix of self-assessment questions and a checklist of recommendations. Not only is it a useful read, but it also emphasises the increased international recognition of the importance of security when it comes to trustworthy AI.
DSIT has said that the AI Cyber Security Code is intended as an addendum to the Code of Practice for Software Vendors – one of a number of modules it developed to set out minimum security expectations in areas where industry is not doing enough to address risk.
This means that, in the UK, when it comes to AI cyber security risk, the Code is now the recommended baseline and should be implemented as a minimum. Whilst it is described as ‘voluntary’, it probably won’t be in practice.
That’s because, whilst this two-part intervention will likely be embraced by those actively looking to understand what ‘good’ AI security looks like, it may in due course be used against those that aren’t. In the event of an incident, regulators and claimants are each likely to turn to the Code, and if you’re not doing the bare minimum expected, then this will undoubtedly count against you.