The shared bargain
Every regulated industry reached the same arrangement with AI before anyone wrote an AI rule. You may use the capability, and in exchange you document the accountability: who owns the decision, what the inputs were, and what happened after. The substance of the obligation barely changes between a bank examiner, a HIPAA audit, and a bar discipline committee. The vocabulary does.
That is the useful insight for a team planning adoption, because it means the compliance work is mostly industry-agnostic. The model is a component. The compliance surface is the system wrapped around it: where data flows, what the system is allowed to do unsupervised, and what evidence it leaves behind. Teams that treat regulation as a reason to avoid AI usually discover the same controls they already run for humans and software would have covered most of it.
What regulators ask about AI
Five questions show up under almost every regime, whatever the statute is called.
Who is accountable?
Every regime expects a named human or body that owns the AI-assisted decision. "The model decided" is never an acceptable answer to an examiner.
Can you show the decision trail?
Inputs, outputs, and the reasoning artifacts around a consequential action, reconstructable months later. Logs that vanish after the run do not count.
Where does the data go?
Residency, retention, and third-party processing. Whether a vendor trains on your data is a contractual question regulators treat as a disclosure duty.
What happens when it is wrong?
A documented path from detected failure to correction, and a person who can halt the system. Undocumented failure handling reads as no failure handling.
Can you stop it?
The ability to suspend the AI function without taking the business process down with it. Runbooks name the person and the procedure.
Industry by industry
Same questions, different rulebooks. Financial services and healthcare link to dedicated pages; legal and insurance follow as those pages ship.
| Industry | What the rules care about | Where AI lands first |
|---|---|---|
| Financial services | Model risk management, GLBA privacy, SOX controls, <a href="/glossary/pci-dss/" target="_blank" rel="noopener noreferrer">PCI-DSS</a> where payments flow. Regulators expect a named owner for every consequential model and evidence it was validated. | Fraud review, underwriting support, document analysis, code and infrastructure work. Anything advisory behind a human decision lands first. |
| Healthcare | HIPAA and the <a href="/glossary/business-associate-agreement/" target="_blank" rel="noopener noreferrer">business associate agreement</a> that follows any vendor touching <a href="/glossary/phi/" target="_blank" rel="noopener noreferrer">PHI</a>. The question is never whether the model is good; it is where the data went and who signed for it. | Admin workflows, documentation, triage support, internal tooling. Clinical decision support carries a heavier approval path and usually belongs later. |
| Legal | Privilege and confidentiality above everything. A vendor that trains on client matter data is a breach of duty, not a feature tradeoff, and bar rules put the responsibility on the lawyer. | Research assistance, contract review, e-discovery triage, drafting support. Work product stays behind the same walls as the documents it came from. |
| Insurance | State-level regulation, actuarial justification for model-driven decisions, and unfair-discrimination rules that apply to the model's output the same as to a person's. | Claims triage, underwriting support, document processing. Decisions that price or deny need a documented rationale a regulator can read. |
The frameworks that actually apply
The named instruments most deployments touch. This maps the landscape; it is not legal advice, and the details move often enough to verify against the source.
HIPAA
US healthcare. PHI handling, minimum necessary, and the business associate agreement that binds any AI vendor touching patient data.
PCI-DSS
Payment card data anywhere in the flow. An AI feature that reads cardholder data inherits the full PCI-DSS scope of the system it sits in.
SOC 2
The trust baseline your customers impose even when no statute does. AI features need to live inside the same control set the rest of the platform attests to. See SOC 2.
GDPR
EU personal data plus a specific rule on automated decisions: individuals can demand human involvement and an explanation when a decision about them is made by machine.
EU AI Act
The EU AI Act tiers obligations by risk. Hiring, credit, insurance, and health-adjacent uses sit in the high-risk band with documentation, oversight, and logging duties phasing in through 2026 and 2027.
Model risk management
The financial-services discipline (SR 11-7 lineage) that expects every consequential model to be validated, monitored, and owned. An LLM is a model under this lens too.
The governance layer that satisfies them
Five controls answer most of the five questions. They are also what we build in an AI governance and risk engagement.
Authority boundary
A written list of what the system may do alone, what needs approval, and what it may never do. The authority boundary is the document an examiner asks for first.
Decision record
Every consequential action logged with its inputs and context. A decision record is what makes a decision trail reconstructable.
Human-in-the-loop
An approval gate on the actions that are irreversible or high-impact. Human-in-the-loop is how "accountable person" becomes a mechanism instead of a policy.
Audit trail
A complete, tamper-evident history of what the system did. The audit trail serves two audiences: your incident review and the examiner's.
Kill switch
A named person and a tested procedure for halting the system. The kill switch is the difference between a bad day and a regulatory event.
Where agentic AI fits
The pattern that survives regulated deployment is bounded autonomy. The agent acts inside a written authority boundary, produces a record for every consequential action, escalates the decisions that cross it, and stops when told. Governed AI is the name we give that arrangement, and it is the only version of autonomy an examiner will accept.
The alternative people actually ship is worse in both directions. Full autonomy in a regulated process is how enforcement actions get written. Full manual operation is the cost AI was supposed to remove. The governed middle is where the workload and the accountability both land, and it is how agentic AI systems get deployed inside financial and healthcare environments without waiting for permission to exist.
Five questions before you deploy
Which decisions does the AI influence?
Influence is enough to create obligations. The inventory comes before the deployment, and "we will figure it out later" is how findings get written.
Who owns each consequential output?
A role, not a team. When the regulator asks who is accountable, a committee name and a job title is the answer they want to hear.
Can you reconstruct a decision from six months ago?
Examinations happen long after the action. If the trail expires with the log retention window, the control never existed.
What does your vendor do with your data?
Training use, retention, subprocessors. The contract answers these or the risk is yours by default, and it travels to the client whose data it was.
Can you stop it without stopping the business?
The halt procedure is only real if someone has run it. A kill switch that has never been exercised is a hope, not a control.
Frequently asked questions
What AI governance requirements apply to financial services companies using LLMs? ▾
The expectations trace back to model risk management rather than any AI-specific statute: a validated model, a named owner, ongoing monitoring, and a documented decision trail. Add GLBA privacy duties for customer data, SOX controls where outputs feed reporting, and third-party oversight for the vendor. An LLM does not get an exemption because it is new; it inherits the framework your existing models already operate under.
Is generative AI allowed under HIPAA? ▾
Allowed inside the same rules that govern any other processor of health information. The vendor needs a business associate agreement, PHI stays inside agreed uses, and access is limited to the minimum necessary. Plenty of healthcare AI deployments are compliant; the ones that fail are the ones that sent patient data to a tool with no BAA behind it.
Can AI agents operate in a regulated environment? ▾
Yes, within boundaries. The workable pattern is governed autonomy: the agent acts inside a written authority boundary, every consequential action produces a decision record, high-impact actions wait for human approval, and a named person can halt it. Agents deployed without that layer are where the enforcement stories come from.
Does the EU AI Act apply to us? ▾
If you offer AI-enabled products or services in the EU market, or your system's output is used there, it likely does. Obligations tier by risk, and hiring, credit, insurance, and health-adjacent uses sit in the high-risk band with documentation and human-oversight duties phasing in through 2026 and 2027. Confirm the current dates against the regulation or counsel, since the rollout has been adjusted before.
What is the difference between AI compliance and AI governance? ▾
Compliance is the external set of rules you must satisfy. Governance is the internal layer that produces the evidence: authority boundaries, decision records, oversight gates, audit trails. Compliance tells you what to prove; governance is how you prove it, and the layer has to exist before the audit, not during it.
Do we need a compliance program before deploying AI? ▾
You need the control layer before the deployment, which is a smaller ask than a full program. Start with the five questions above: decision inventory, named owners, a reconstructable trail, vendor data terms, and a tested halt procedure. The formal program can grow around those, and building them first is how the formal version ends up describing something real.
Deploying AI inside a regulated environment?
A thirty-minute call is usually enough to tell which controls your situation actually needs, and which requirements apply to your case at all.