Responsible AI governance on SAP BTP — compliance, audit trails, and the human-in-the-loop requirement

Accountability does not delegate

Start with the legal fact that frames everything else. Under GDPR, the controller “shall be responsible for, and be able to demonstrate compliance with” the data-processing principles — the accountability principle. Saudi Arabia’s PDPL takes the same position: controllers “are held ultimately accountable for the processing undertaken”, and embeds accountability among its seven data-protection principles — “appropriate measures and records in place to be able to demonstrate your compliance”. An agent may act autonomously; your organization answers for it. That is why AI governance on BTP is a board-level design question, not a technical afterthought.

What SAP itself does — a useful template

SAP’s internal AI governance is well documented and worth studying, because it shows what a mature regime looks like. SAP maintains a Global AI Ethics Policy governing “the development and deployment of SAP’s AI systems in line with the established guiding principles and core organizational values”. The 2023 AI Ethics Handbook enumerates seven guiding principles — including “We design for people”, “We strive for transparency and integrity” and “We place data protection and privacy at our core” — with “Human Agency & Oversight” as the first pillar of the policy; in September 2024 SAP updated the policy to ten guiding principles grounded on the UNESCO Recommendation. (Two versions, deliberately kept distinct here: seven principles in the 2023 handbook, ten in the 2024 update.)

Three mechanics in SAP’s regime translate directly to what customers should build:

• A hard human-intervention requirement. SAP’s product standard states it in five words: “Allow human intervention for all automated decision processes.”
• A high-risk gate. High-risk use cases “must first go through an assessment process of the AI Ethics Steering Committee before they can be further developed, deployed, and sold” — and SAP’s high-risk criteria explicitly include fully or partially automated decision-making where “no human intervention or human supervision takes place”.
• An escalation chain with named bodies. Classification is checked by the Global AI Ethics organization; confirmed high-risk cases go to the steering committee, supported by an external AI Ethics Advisory Panel and internal steering committee.

If SAP requires a committee gate before its own high-risk AI ships, your organization deploying agents against payroll, payments or personal data deserves no less.

The regulatory floor: GDPR, the EU AI Act, and PDPL

GDPR sets the data-handling baseline for any agent touching personal data: processing must be lawful, fair and transparent; data must be collected for specified, explicit and legitimate purposes and limited to what is necessary; and it must be kept no longer than necessary and processed with appropriate security. For agent design, purpose limitation bites hardest: an agent grounded on a broad document store must still be able to show that each category of personal data it reads serves the stated purpose.

The EU AI Act makes human oversight a design property, not a policy statement: high-risk AI systems “shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons”, including the ability to intervene or interrupt “through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state”. These obligations phase in through 2026–2027 — forward-looking for many deployments, but the design bar is already clear.

Saudi Arabia’s PDPL matters to every organization running SAP workloads touching the Kingdom: approved by Royal Decree No. 19/M, in force since 14 September 2023, it applies extraterritorially — to processing of personal data of individuals in the Kingdom “by any means from any party outside the Kingdom”, and grants data subjects rights to be informed, access, correction, destruction, and withdrawal of consent. An agent that has ingested personal data into its grounding store must be able to honor a destruction request — a retrieval-architecture decision you make on day one, not at the first complaint.

Designing the human-in-the-loop — and the audit trail

How we design this at IOTEK — our delivery practice, aligned to the sourced requirements above. Escalation design starts by classifying every agent action into three tiers: advise (agent drafts, human acts), act-with-approval (agent prepares the transaction, a named human role approves before commit), and act-with-review (agent commits low-risk actions, humans sample-review). Anything touching personal data, payments or the ledger sits in the first two tiers; the EU AI Act’s stop-condition thinking applies as a standing capability — a way to halt the agent safely without halting the business process.

On audit trails, an honesty note our readers should hold every vendor to: at the time of writing we could not verify SAP AI Core-specific audit-logging capabilities from fetchable SAP documentation, so this article makes no claim about them — verify the current BTP Audit Log and AI Core documentation directly during your design phase. What the regulations require is clear regardless: GDPR’s accountability principle and PDPL’s records requirement mean you must be able to demonstrate compliance. Our practice is to design the evidence layer ourselves rather than assume it: log every agent decision with its inputs, grounding sources, output and approver; make the log readable by auditors, not just engineers; and test the “show me why the agent did that” question before go-live, because a regulator will eventually ask it.

Agents change who executes; they do not change who answers. Build the committee gate, the three-tier escalation, and the evidence layer before the first agent touches production — because under GDPR and PDPL alike, “the AI did it” is not a defense, and boards that treat oversight as a design requirement will move faster, not slower, when regulators come asking.

Sources