AI-Assisted Policy Development: Keeping Humans Responsible for Data Citizens

Key Takeaways

• The Necessity of the “Audited Voice”: Governance professionals must independently justify every control implemented to mitigate audit risks effectively.
• Mitigation of AI Hallucination Risks: Relying on generative AI without oversight risks fabricating citations; professionals must verify sources rigorously.
• Implementation of a Core Data Model: Effective policy management needs a solid framework linking principles, controls, procedures, evidence, and KPIs.
• Continuous Validation and Decay Management: Policies decay; developers must implement validity checks to ensure alignment with regulations.
• Utilisation of Quality Control Dashboards: Organisations should use dashboards to identify compliance anomalies like regulatory orphans and evidence phantoms.
• Preservation of Human Intuition via “Time Boxing”: Document human perspectives before using AI tools to prevent bias and ensure accountability.
• Strategic and Constrained AI Utility: AI tools like NotebookLM and Claude enhance research but require careful integration and evaluation.
• Organisational Change Management: Implementing a “day in the life” methodology clarifies compliance responsibilities for operational roles.

Webinar Details

Title: AI-Assisted Policy Development: Keeping Humans Responsible for Data Citizens
Date: 2026-05-21
Presenter: Howard Diesel
Meetup Group: African Data Management Community
Write-up Author: Howard Diesel

What Risks Exist with AI in Policy Development?

Howard Diesel opens the webinar and begins his critical examination of the risks associated with AI-assisted policy development, referencing a controversial national AI policy that contained fabricated citations. Specifically, six out of 67 academic references were entirely AI-generated and non-existent, resulting in the termination of the policy’s authors. This incident underscores the necessity for rigorous human oversight in policy creation.

Despite these risks, the speakers advocate for utilising controlled AI environments, such as NotebookLM, to accelerate initial research by generating mind maps and summaries from strictly defined source materials. This methodology significantly reduces the time required to review global standards while mitigating the risk of hallucinations.

Figure 1 AI-Assisted Policy Development Deck

Figure 2 Source / Reference (Official Title)

Figure 3 NotebookLM: Saudi National AI Strategy

Figure 4 NotebookLM Generated Mind map

What is the Importance of a Core Data Model?

A foundational “Core Data Model” is presented as a structural imperative for policy systems. This model categorises key entities, including principles, thematic categorisations, controls, standard operating procedures (SOPs), evidence requirements, roles, management intent, and key performance indicators (KPIs). To avoid overwhelming organisational units with excessive individual controls, governance professionals are advised to consolidate them into a manageable suite of eight to ten SOPs.

Furthermore, a robust traceability system is required to validate that every control legitimately originates from a verified regulatory instrument. This structured architectural approach prevents critical errors, such as relying on AI to generate citations, which can inadvertently misquote authors and damage professional reputations.

Figure 5 Policy Inspector System Mind map

Figure 6 Core Data Model

How should Controls be Consolidated in Governance?

Navigating the extensive, interconnected data inherent in policy management necessitates dedicated infrastructure, such as internal policy libraries. When developing policies for clients, integrating “management intent”—the specific operational objectives an organisation aims to achieve—transforms a generic governance starter kit into proprietary intellectual property.

To preserve the integrity of this data, developers must implement strict validity pipelines. This protocol includes monitoring the decay dates of source documents and verifying official versions to ensure that organisational policies remain aligned with current, rather than obsolete, regulatory instruments.

Figure 7 Policy Library

Figure 8 Glossary

Figure 9 Skills Development: Step One – Policy Framework Research

Why are Structured Policy Models Essential in Regulation?

The necessity of structured policy models is particularly pronounced in heavily regulated environments, such as the banking and insurance sectors. Policy developers must navigate stringent, overlapping regulations, such as the cybersecurity mandates enforced by central banking authorities like SAMA.

When constructing automation models for insurance claims, professionals must simultaneously map complex business processes against regional directives, such as Treating Customers Fairly (TCF). Establishing robust foundational data and AI policies is a critical prerequisite before any automated business processes or generative AI techniques can be safely deployed in these sensitive sectors.

How should Organisations Ensure Continuous Compliance and Governance?

Policies operate in a continuous state of decay as external regulations and internal enterprise architectures invariably evolve. Consequently, organisations must maintain an ongoing review process to ensure their governance frameworks remain applicable and compliant. A central tenet of this continuous compliance is the cultivation of an “audited voice”.

Governance professionals must be capable of independently articulating the lineage, provenance, and operational justification for every implemented control and standard operating procedure. In the context of an official audit, attributing policy decisions or the derivation of evidence to artificial intelligence is unacceptable and poses significant professional risk.

Figure 10 CCAA Findings

Figure 11 Driving the Policy

Figure 12 Governance Framework

How can Dashboards Prevent Compliance System Failures?

To pre-empt systemic compliance failures, developers must utilise warning dashboards designed to identify critical discrepancies within the policy framework. These dashboards track anomalies such as “regulatory orphans” (controls lacking an originating mandate), “evidence phantoms” (required artefacts that do not currently exist), and “role friction” (assigning responsibilities to roles absent from the organisation’s specific operating model).

Maintaining a comprehensive evidence catalogue is essential; this repository must explicitly define each artefact, designate a responsible owner, and specify retention periods. Additionally, mapping internal KPIs to deployed controls allows organisations to empirically measure the policy’s functional efficacy.

Figure 13 Diagnostic

Figure 14 Skills Development: Step Nine – Compliance & Implementation

Figure 15 Evidence Catalogue

Figure 16 KPI Name

How does AI Integration Impact Policy Development Processes?

Integrating AI into policy development requires substantial iteration and demands rigorous quality-control evaluations to verify schema integrity, control coverage, and overall traceability. Users must frequently refine their prompts to explicitly instruct AI models, which can occasionally necessitate complete procedural restarts when a large language model exhausts its context window.

Furthermore, successfully operationalising these frameworks demands significant organisational change management. Governance professionals are advised to utilise a “day in the life” methodology to clearly delineate the specific compliance workloads and periodic operational responsibilities expected of individual organisational roles.

Figure 17 Skills Development: Step Six – List of SOPs

Figure 18 Skills Overview

Figure 19 Roles & Responsibilities

How can we Ensure Human Accountability in AI?

Howard concludes by affirming the indispensable role of human intuition in AI-assisted workflows. Because large language models deterministically generate probabilistic answers, they can confidently fabricate information if left unchecked. To mitigate this phenomenon, professionals must manually verify every source, read academic abstracts, and substantiate all quotations.

A highly recommended strategy is to employ “time boxing,” in which practitioners document their independent human perspectives before consulting artificial intelligence. This exercise prevents human judgment from being subconsciously biased by algorithmic autosuggestions, ensuring that the final policy architecture remains fundamentally grounded in human accountability and verified regulatory frameworks.

Figure 20 Incident Report One

Figure 21Policy Description

Figure 22 Operational Protocol

Figure 23 Discipline Deep-dive