AI Governance and the Unsolved Inference Problem

Your AI knows more than it should, and there's no way to make it forget.

Most companies think of AI data leaks as a preventable engineering problem: block the wrong outputs, filter sensitive content, and you're safe. But academic research tells a different story. The tension between an AI model's usefulness and its confidentiality isn't a gap to patch, it's an information-theoretic dilemma. And it affects every enterprise deploying AI today.

This isn't hypothetical. Samsung, ChatGPT, and GitHub Copilot have all been at the center of documented data breaches. But those visible incidents are just the surface of a much deeper structural problem.

The regulatory landscape: a patchwork in motion

The EU AI Act (Regulation 2024/1689) entered into force in August 2024 and will be fully applicable by 2027. It follows a risk-based approach: banned practices like social scoring have been prohibited since February 2025, high-risk AI (credit scoring, recruitment, healthcare) faces strict conformity requirements from August 2026, and General-Purpose AI models like GPT and Claude must meet transparency obligations from August 2025. The accompanying GPAI Code of Practice was published in July 2025 and signed by OpenAI, Anthropic, and Google. Violations can result in fines of up to 35 million euros or 7% of annual revenue.

Meanwhile, the rest of the world is diverging. The US under President Trump revoked Biden's AI safety order in January 2025, prioritizing innovation over compliance. China requires all AI-generated content to be labeled, both visibly and in metadata, since September 2025. Singapore published the world's first governance framework for Agentic AI in January 2026.

Two enterprise standards are emerging as critical: the NIST AI Risk Management Framework (AI RMF 1.0) and ISO/IEC 42001:2023, the world's first certifiable AI management system standard. Microsoft and KPMG International have already been certified. But a Deloitte study reveals the gap: only 9% of organizations have a mature AI governance framework, and just 28% have board-level AI oversight.

The incidents that should keep you up at night

In March 2023, less than a month after Samsung Semiconductor gave engineers access to ChatGPT, three separate incidents occurred. One employee pasted proprietary chip measurement source code into ChatGPT. A second submitted yield optimization code. A third transcribed a confidential meeting and asked ChatGPT to generate minutes. Samsung's response was blunt: the leaked data was no longer deletable from OpenAI's servers. They initially limited prompts to 1,024 bytes, then banned all generative AI tools on company devices entirely.

That same month, a Redis bug at OpenAI exposed conversation titles and payment data of other ChatGPT Plus subscribers for nine hours, affecting 1.2% of all Plus users. In June 2025, the CamoLeak vulnerability (CVSS 9.6) in GitHub Copilot allowed invisible HTML comments in public repos to exfiltrate source code from private repositories. And Lasso Security discovered that caching mechanisms in Microsoft Copilot had exposed private GitHub repos from 16,000 organizations, including IBM, Google, and PayPal.

The scale is staggering. Cyberhaven's Q2 2025 analysis of 7 million workers found that 39.7% of all AI interactions contain sensitive data, a 270% increase since March 2023. Generative AI tools cause 32% of all unauthorized data movements in enterprises. And 71.6% of generative AI access happens through personal, unmanaged accounts.

Why inference attacks are the real governance problem

The incidents above involve direct data leaks, the model outputs confidential information verbatim. But the deeper problem is indirect information disclosure through inference.

Membership Inference Attacks determine whether specific data was in a model's training set. Model Inversion Attacks reconstruct training data from model outputs, researchers showed in 2015 that recognizable photos of training subjects could be generated from a facial recognition model's confidence scores. Property Inference Attacks extract properties of the training data distribution, like gender ratios or disease prevalence in a fine-tuned medical model.

Researchers at ETH Zurich demonstrated something even more concerning: the safety mechanisms themselves create side channels. A memorization filter that prevents a model from outputting training data paradoxically becomes a binary oracle, if the system does output a text, that text was guaranteed not in the training data, enabling near-perfect membership inference through systematic querying.

The practical implication is stark. Imagine asking an LLM fine-tuned on multiple companies' data: \"Is my pricing more competitive than Company X's?\" Through a series of yes/no comparison questions, an attacker can reconstruct relative values, a binary search on confidential pricing information. Even without explicit confidence scores, response latency, hedging language, and answer patterns reveal implicit certainty levels.

The mathematical foundation for this was established in 2003 by Dinur and Nissim: the Fundamental Law of Information Recovery proves that answering a sufficient number of seemingly harmless queries about a database inevitably reveals sensitive information. This isn't an implementation gap. It's a theorem.

The four defense lines, and their honest track record

Differential Privacy offers the strongest mathematical guarantees. Google's VaultGemma 1B (September 2025) shows no detectable memorization. But DP only protects training data, not data fed at inference time through context, RAG, or system prompts.

Federated Learning prevents raw data centralization but doesn't protect against inference attacks. Gradient leakage attacks can reconstruct training data from shared gradients. FL actually expands the attack surface.

Machine Unlearning, selectively \"forgetting\" data, has fundamentally failed. CMU researchers showed that all common unlearning methods can be bypassed by \"relearning attacks.\" After a model \"forgot\" Harry Potter texts, fine-tuning on publicly available GPT-4 summaries caused it to output verbatim memorized passages from the novels.

Guardrails (NVIDIA NeMo, LLM Guard, Azure Content Filtering) catch obvious leaks but fail systematically against indirect inference and multi-turn attacks. In August 2024, a red team attack on Slack AI extracted API keys from private channels through instructions injected into files.

What this means: governance is the only realistic strategy

No single technology solves the inference problem. The core paradox remains: for an AI model to be useful, it must encode knowledge from data; for it to be private, it must not reveal that knowledge when queried. In neural networks, knowledge isn't stored in discrete, lockable memory cells, it's distributed across billions of parameters.

The practical consequence for enterprises is a defense-in-depth approach:

Data governance first. Define which data is allowed to flow into which AI systems before any model touches it. This is the most effective control, and the most overlooked. A governance layer that intercepts, classifies, and enforces policies on every interaction between your people and AI models prevents the problem at its source.

Strict tenant isolation. Separate RAG indices, tenant-specific model instances, and never share fine-tuned models across organizations. Microsoft's own Azure OpenAI documentation warns explicitly against this.

Continuous red-teaming. Guardrails are necessary but insufficient. Regular adversarial testing, including inference attacks, multi-turn probing, and side-channel analysis, is the only way to understand your actual exposure.

Treat compliance as a floor, not a ceiling. The EU AI Act and ISO 42001 create necessary governance structures, but they don't explicitly address the inference problem. Companies that treat certification as \"done\" are the ones most exposed.

The question isn't whether your AI model will reveal confidential information. It's how much, and whether the residual risk is acceptable for your use case.

At hegyon.ai, we build the governance layer that sits between your people, your data, and AI models. Every interaction is governed, every data flow is controlled, and every query passes through policy enforcement, because in the age of inference attacks, perimeter security alone isn't enough.