Who Owns AI Compliance? Hidden Risks and Practical Challenges Organisations Are Starting to Face
Who, inside a company, is responsible for AI compliance?
It sounds like a simple question, until you realise how many AI systems a company may already be using without fully knowing it.
This was the starting point of our recent Women in Digital Forum webinar, delivered as part of TWG6’s ongoing work on women and the AI economy. The session featured Dr. Rimma Dzhusupova, AI compliance expert, researcher at Eindhoven University of Technology, and founder of ExploreRay, who brought fifteen years of experience leading global AI deployment in the oil and gas industry.
The AI you do not know you have
The first obstacle to compliance is inventory. When Dr. Dzhusupova asks organisations how many AI systems they use, the answers range from five to fifty, and sometimes she is told the organisation uses none at all, but that that is almost never true. Microsoft Copilot, Adobe’s AI assistant, Oracle HR platforms, Power BI’s regression tools, all these tools are all AI systems under the EU AI Act’s definition, and embedded in the daily operations of most large enterprises. The top management of those organisations frequently has no visibility over how these tools are being used at department level.
This matters because the AI Act does not regulate organisations. It regulates specific systems and the roles people play in relation to them. Every organisation using an AI system is either a provider (the entity that places it on the market or puts it into service) or a deployer (the entity that uses it under its own authority). This distinction is, however, not fixed and shifts depending on how a tool is used.
“If you don’t develop AI, AI Act doesn’t apply to us. It’s only for providers. But AI compliance affects everybody.”
The hidden provider trap
Dr. Dzhusupova walked through a scenario that she has encountered repeatedly in her consultancy work. An organisation purchases an HR platform from a vendor. The vendor later introduces an AI assistant that can summarise employee performance assessments. HR adopts it enthusiastically. The use case of AI-assisted evaluation of employees is classified as high risk under the AI Act, because it directly affects fundamental rights.
The vendor, when asked whether it will take on the responsibilities of a high-risk AI provider, declines. In Dr. Dzhusupova’s experience, this is the norm rather than the exception. Vendors consistently argue that how a customer applies their tool is the customer’s responsibility. The organisation, which simply wanted a productivity feature, has now become the provider of a high-risk AI system. It must produce technical documentation, conduct a risk assessment, complete a conformity assessment, and obtain CE marking.
If the vendor will not cooperate and the organisation cannot produce the required evidence, the only remaining option may be to stop using the tool entirely, after it has already been deployed at enterprise scale.
The lesson Dr. Dzhusupova drew from these cases is contractual: before signing a multi-year enterprise software agreement, organisations must establish in writing what will happen if a use case becomes high risk, who will bear the provider responsibilities, and what technical documentation the vendor is obligated to supply.
“Most of the time it’s really a silent shift. Most of the organisation doesn’t know about these things. And that’s what scares.”
Who owns it internally
The second problem is that AI compliance sits between functions without belonging to any of them. Legal does not have the technical knowledge, IT does not understand operational use, operations does not want to be interrupted and procurement considers its job complete once a contract is signed. The result is that no single person or team has sight of what AI is in use, how it is being used, or whether any of it is generating high-risk exposure.
What is needed is a cross-functional AI governance function that actively maps AI use across every department, assesses the risk profile of each use case, establishes internal policies that define where specific tools may and may not be applied, and monitors compliance on a continuous basis. Internal policies, Dr. Dzhusupova noted, are themselves a meaningful compliance signal: an external auditor finding an enforced policy prohibiting high-risk AI use cases is a substantially different situation from finding no policy at all.
Gender and AI adoption
Research suggests that women use AI tools at significantly lower rates than men in professional settings, with some studies indicating a ratio of roughly one to three. One explanation offered is that women are more likely to experience AI-generated output as a form of cheating.
Dr. Dzhusupova observed that from a bias perspective, greater AI adoption among women could be beneficial: a well-governed CV screening tool that removes gender from the evaluation process creates conditions for fairer assessment. The moderation team noted a further distinction that the data rarely captures and its the difference between use and adoption. An employee who opens Copilot once and an employee who has integrated it fluently into their workflow register identically in usage statistics. Understanding which is which is essential to designing AI literacy programmes that shift behaviour, and it is a question TWG6 intends to pursue.
The compliance deadline is closer than most organisations think
The high-risk AI provisions of the AI Act apply from December 2027. Dr. Dzhusupova’s assessment of industry readiness is that awareness remains close to zero in most sectors, and the infrastructure for compliance is not yet in place. In the Netherlands alone, only two conformity assessment bodies currently exist to serve the entire market. The organisations that begin building AI governance functions, conducting use case inventories, and reviewing vendor contracts now will be in a fundamentally different position from those that wait.
For the Women in Digital Forum, it is a workforce challenge, a leadership challenge, and a gender challenge. The question of who owns compliance inside organisations will increasingly determine who shapes AI adoption.
“If I look on the industry, I’m really wondering how it’s going to be next year, because most of the industry are still sleeping.”
