AI Governance for Data Managers

Executive Summary

This webinar covers the key themes in AI governance and data quality management and how they play out across different industries. Howard Diesel makes the case for Total Quality Management in manufacturing and for applying quality management principles to data mining in industrial engineering. It also draws out the distinction between governance and implementation frameworks in AI development, and works through what it actually takes to build a good AI chatbot. Howard covers the collapse of multi-party computation and model degradation too, proposing evaluation-driven development as a way through both, along with strategies for improving AI systems: fraud detection, accountability, baseline measurement, and how AI techniques apply to insurance claims processing and data analysis.

Webinar Details

AI Governance and Data Quality Management

Howard Diesel opened the webinar by noting this series is part of a broader initiative tied to a new white paper he wrote on how different facets of AI governance fit together. In recent training sessions, he’d noticed participants were genuinely uneasy about data quality, particularly around LLMs and their predictive outputs, which raises a real question: are we starting to care more about whether results look reliable than whether the input data actually is? Getting AI governance right means finding a genuine balance between data quality and output reliability, not favouring one over the other.

That balance points back to principles Shewhart and Deming laid out decades ago under Total Quality Management (TQM), even though neither of them used the term “data quality” specifically. Howard’s view is that introducing the newer term has, somewhat ironically, weakened people’s grasp of the older TQM principles. A conversation with an expert in Learning Management Systems and chatbots brought him to the idea of “evaluation-driven development,” which looks like a promising way to fold evaluation-driven thinking back into TQM and strengthen both data quality and AI governance in the process.

Figure 1 Total Quality Management and Evaluation-Driven Development

Implementing Total Quality Management in Manufacturing and AI

Total Quality Management is fundamentally about improving manufacturing processes to get a better product out the other end. W. Edwards Deming’s 14 points are central to this, pushing for continuous quality checks throughout production rather than just inspecting at the end. Deming’s argument was that fixing defects after the fact is expensive, so it’s better to improve and standardise raw materials from the start. Keep the inputs and processes consistent, and the outputs end up reliable and uniform too.

In today’s fast-moving world of information and AI products, quality matters a lot more than sheer data volume. That means genuinely engaging with consumers to understand what they actually need, in the spirit of Deming’s own philosophy of building products around customer expectations. Deming’s own turnaround of a struggling fast food business is a good example of how much a customer-driven approach can improve both market share and product quality. Prioritising quality and customer engagement together tends to drive both innovation and sustainable growth.

Howard drew on lessons from Japanese manufacturers, starting with customer interviews to understand what people actually wanted. He traced the poor taste of the burgers back to meat supply contracts awarded mainly on price rather than quality. Restructuring those contracts and bringing in a Plan-Do-Check-Act framework helped improve food quality, and it all came back to starting with the consumer’s actual needs rather than assuming you already know them.

The insurance industry faces a similar balancing act between business efficiency and customer benefit, particularly in the First Notice of Loss (FNOL) process, where AI is increasingly used to streamline operations and reduce the need for human involvement. Legal counsel advising on EU AI regulations has pointed to the need to genuinely integrate business requirements with consumer needs, which takes real planning, requirements analysis, and quality control to make sure products actually meet expectations. Quality assurance is what confirms that materials and processes stay within tolerance, which is what makes the whole thing effective.

Developing the product properly means running root cause analysis on failures from both an assurance and a control angle. That collaborative approach is what lets quality monitoring get integrated across every part of the process, so the end product actually meets customer expectations. A common misconception is that data quality only concerns the raw materials and inputs going into an AI model, but it covers a lot more ground than just that.

Getting a full picture of quality means looking past data quality alone to total quality management more broadly. That wider view puts weight on the need to hold high standards for every input, whether it’s a database table, a document, or a video, so the data feeding models and systems stays high quality throughout. Shifting the conversation from data quality alone to total quality management builds a stronger foundation for everything downstream.

Figure 2 Total Quality Management

Figure 3 Total Quality Management pt.2

Quality Management and Data Mining in Industrial Engineering

TQM is still a core principle in industrial engineering, even though a lot of large corporations now use their own terminology for essentially the same idea. A recent conversation with a Learning Object Model (LOM) developer made clear how important it is to keep the model aligned with what consumers actually want, high-quality data needs to sit alongside functional efficiency, not instead of it. The developer’s work with a marketing firm added a creative dimension too, shifting the focus from just minimising inaccuracies toward genuine innovation in the model. Howard’s point was that quality management practices keep evolving, and building effective models means striking a real balance between creativity and precision.

There’s a newer approach to managing quality metrics in LLMs that pushes back against traditional methods by putting individual concerns about data completeness and consistency first. Howard argues that the conventional definition of a ‘good’ LLM tends to overlook flexibility in the data quality dimensions, and that rigidity can stifle innovation. He’s been exploring alternative methodologies within an AI implementation framework, making the case for a more nuanced view of what quality metrics should capture. Fostering that kind of innovation in LLMs means moving away from rigid quality standards toward something more adaptive, one that can accommodate concerns as they evolve.

CRISP-DM, the long-standing data mining framework, gets criticised for its waterfall approach: sequential phases moving through business understanding, data exploration, data preparation, modelling, and evaluation. One real problem is the tendency to circle back to earlier stages, business understanding especially, whenever issues crop up, which slows deployment down. Critics argue it doesn’t build in enough agile practice, which is exactly what would make data mining more responsive and iterative.

CRISP-DM++ has emerged as an attempt to move away from that waterfall approach, using multiple iterations to refine the model, collecting data, identifying key features, working toward an acceptable level of error reduction before deployment. Agile principles are still hard to build in properly, though, since the process tends to lean on heavy modelling effort rather than the kind of flexible, iterative cycles agile is meant to bring.

Figure 4 AI Implementation Framework for Total Quality Management

Figure 5 AI Implementation Framework for Total Quality Management pt.2

Evaluation-Driven Development and Quality Management in Agentic AI

Evaluation-driven development matters a great deal in Agentic AI and Learning Resource Management Systems specifically. It prioritises building a prototype fast, often with no-code techniques, cutting development time and getting it in front of users quickly. From there, gathering feedback becomes the priority, to actually understand what users need. Setting performance metrics this early is genuinely hard, but the iterative process keeps improving based on feedback, which produces a development environment that’s adaptable and stays centred on the user.

Evaluations, in this context, mean building an observability platform that can assess both online and offline interactions. Online evaluations happen live, as the system makes recommendations and captures real-time feedback; offline evaluations analyse traces of past interactions between users and the system, generating feedback from that review. LLMs themselves get used as evaluators here, judging response quality and factors like toxicity, checking, for instance, whether the agent is communicating negatively or drifting from the standards it’s meant to follow.

Building a real-time Line of Moderation system is genuinely complex, particularly around managing toxicity checks on agent responses. The evaluations approach helps here, focusing on how to assess and monitor these systems while they’re in production and making sure alerts actually reach users and developers. Combined with total quality management principles, this eval-driven approach is a promising way to improve software development more broadly.

The implementation framework for AI quality covers data quality dimensions like accuracy, timeliness, completeness, and representational accessibility, plus AI-specific criteria like objectivity, believability, and reputational risk. Objectivity is about whether the model is actually free of bias; believability is about whether outputs are reasonable and explainable, especially when something anomalous shows up. Using the right amount of data matters here too, particularly given the EU AI Act’s emphasis on having enough data to make sound judgments, which sits in tension with GDPR’s push to minimise data usage in the first place.

Pulling data mining, LLMs, and generative frameworks into one cohesive approach means reconciling some genuinely contradictory elements. That takes a comprehensive view of total quality, one built around a systematic set of dimensions that actually capture what users want and how good their input is. Get that integration right, and you end up more responsive to user needs and more effective overall.

Figure 6 AI Implementation Framework for Total Quality Management pt.3

Figure 7 AI Implementation Framework for Total Quality Management pt.4

Understanding the Differences between Governance and Implementation Frameworks in AI Development

The distinction between a governance framework and an implementation framework matters a lot in AI governance. A governance framework is about how to set up a data governance programme: who the stakeholders are, what needs considering. An implementation framework is about actually delivering a successful AI model product, making sure the right information products get used. It’s worth not conflating the two, since they serve different purposes at both the enterprise and project level. Building the Gentek AI also surfaced a related issue: non-deterministic agents behave quite differently from deterministic software, and that difference shapes development in its own way.

In traditional software development, test-driven development gives you test cases to evaluate performance, which is fairly straightforward for deterministic unit tests and integration testing. That changes a lot with LLMs and their agents, which are inherently non-deterministic and can take any number of conversational paths. Defining what a conversation even looks like for an LLM is hard: over-structure it, and you end up with a rigid chatbot experience that might not actually address what the user needs, which just leads to frustration. It’s a real illustration of how complex LLMs and conversational AI get once they’re built into actual software.

Improving an agent depends heavily on getting better, more extensive data, which is quite different from traditional software testing. With a BI product, quality gets defined, tested, and confirmed before deployment; AI systems, agents and LMS platforms included, need enough data and understanding just to be implemented properly in the first place, which means results often only become visible once that data is actually there.

Figure 8 Implementation Framework and Governance Framework

Figure 9 AI Building Challenges

Considerations in Building and Implementing AI Chat Bots

A handful of challenges keep coming up with LLMs: hallucination, accuracy, token usage costs, response latency. A conversation with an education department in Saudi Arabia, for instance, surfaced real delays in chatbot response time that were frustrating users. One fix has been having the chatbot show it’s “waiting,” just to manage expectations. But the black-box nature of LLMs makes it genuinely hard to understand their performance and failures, which is exactly why more transparent, effective approaches to building and deploying them matter.

These challenges get compounded further by what’s called “collapse,” where improvements in an LLM suddenly plateau or even reverse. Scope creep, legacy system integration, and PII security risks all contribute, especially where a chatbot is collecting sensitive data. There’s real uncertainty around how that PII gets managed, and how visible it is compared to a traditional database. All of which points to a need for real change management, and probably a rethink of the existing rules frameworks.

Integrating AI into existing frameworks is genuinely hard, particularly in domain-specific areas like marketing, where personal data considerations complicate adapting AI to established expectations. Standard metrics like hallucination and toxicity often don’t capture the nuanced issues each domain throws up, which means practitioners end up retrofitting data quality to the model’s needs after the fact rather than before. That’s exactly why consumer-defined quality standards matter so much, so the data actually aligns with the rules and supports what the AI is meant to do. A thoughtful approach to integration is what makes the difference in any given field.

Defining quality standards for data entering the system means only accepting data that actually meets them. Key checks should look for bias and PII in incoming data specifically, and during user or customer interactions, it’s worth staying alert to potential issues and having strategies ready to address them. Keeping the right data quality rules in place is what prevents problems and keeps the data reliable overall.

Collapse of the Multi-Party Computation and Evaluation-Driven Development

Getting an LLM to succeed depends a lot on striking the right balance in training, and not over-restricting it. Overtrain a model to minimise toxicity, hallucination, and PII issues, and it can slip into a “dumb mode,” giving less relevant, less useful responses. The fix is putting fewer constraints directly on the model and instead refining the inputs and context it works with. That approach tends to improve interactions while keeping the model’s actual capabilities intact.

That same need for balance came up again during a recent workshop, where an attendee thanked the presenter for the session and confirmed an earlier question had been answered, including one about an upcoming session on building agents with AWS scheduled for mid-month. They connected their question to another participant’s earlier one about evaluation-driven development, asking for related research materials, and the speaker confirmed they had a full presentation and references to share. It was a friendly exchange, with a few clarifications along the way to make sure everyone was on the same page.

Application of Rubrics in Quality Systems

Working with Jonathan, Howard shared that they’re defining a user quality standard flexible enough to vary by individual preference, since some users want more creative “hallucinations” in the output while others want as few as possible. He calls this quality framework a rubric, a term familiar from educational settings, and it’s proving to be an effective assessment tool. An Excel evaluation document has been built to support the process.

Building on that, Howard shared a comprehensive total quality rubric that ties business outcomes to customer benefits, aiming to catch every fraud case without sacrificing value to the customer. It captures the essential process steps and evaluation criteria. He built the initial framework on total quality management principles, then layered in additional requirements from HIPAA standards, ISO 42000, and Risk Management Framework guidelines, refining the rubric iteratively to close gaps and fold quality measures properly into the process.

Figure 10 The 7-Step TQM + EDD Cycle

Figure 11 TQM Rubric

Figure 12 HIPAA Additions

Figure 13 NIST AI RMF Rubric

Process of Analysing and Improving Automated Responses

The core process is plan, do, study, act, with root cause analysis running through it. Howard’s study aimed to check whether the model’s responses actually followed the right path and gave adequate information. Understanding customer needs through voice-of-customer interviews and surveys is a key step, checking whether the service delivers value and surfacing any frustrations users have with the chatbot. Recent feedback from a banking institution flagged some areas worth engaging with and improving.

Current practices can affect customer experience quite a bit, particularly in fraud prevention. Some companies have automated fraud validation to cut the cost of human review, often through automated calls asking customers to confirm a transaction, which can feel cold and unsettling. Preventing fraud is the goal, sure, but talking to a machine instead of a person can be a genuinely unpleasant experience for customers. Which is why customer engagement and communication ought to come first, before automated systems get bolted onto the transaction validation process.

Figure 14 Define Customer Needs and Quality Requirements

Development of Comprehensive Evaluation Rubrics

Improving the efficiency of the claims process starts with clear business objectives and identifying the quality characteristics that actually affect customer satisfaction. That includes measurable quality targets for the normal AI-handled detection scenarios, which usually end in either approval or escalation to a human. Bringing a human into the loop, though, can add delays, sometimes stretching response time to 3-4 days, which is a real problem in time-sensitive situations like burial arrangements. That’s why defining SLAs for timely responses matters, along with making sure human evaluators explain their decisions within a reasonable window.

Getting there means building a comprehensive evaluation rubric, starting with metric standards tied to business requirements and customer value. Compliance frameworks need considering too: HIPAA for data privacy and AI systems, risk management frameworks like AI RMF, EU AI Act applications. A thorough review of the compliance rules covers all the necessary ground, and document management principles matter for insurance claim submissions specifically, records like photographs and police statements all feeding into a rubric robust enough to meet every required standard.

Addressing the challenges LLMs bring is an ongoing job that needs continuous observation and adjustment. A scoring rubric for evaluating how accessible and retrievable essential records are surfaces real concerns about error rates and completeness. A1 covers the difficulty of retrieving crucial records, for instance, while A5 sets the standard that 100% of records need to be quickly retrievable. Governing AI systems well really depends on being able to access submitted records and LLM responses, which is why keeping three years of documentation and making sure the right decision-making records are easy to find matters so much for the reliability and accountability of how LLMs are run.

This AI system keeps detailed records of claim rejections, making sure every submitted document stays accessible for review. It follows different retention policies that dictate when documents get disposed of, and it adheres to compliance ratios and completeness standards. Key metrics, including minimum and maximum guidelines, get closely monitored for data quality, which is what lets the system track and manage claims and compliance information properly.

Figure 15 Define Metrics, Standards and Rubric

Figure 16 EU AI Act and ARMA

Figure 17 Business Outcomes

Measurement of Baseline Performance in Claims Detection

Baseline performance is a key metric for evaluating a model before it goes live, since it establishes the relationship between the features that actually drive outcomes, claim fraud detection among them. Earlier discussions of FNOL and anomaly detection identified claim amount, claim age, and number of previous claims as the features that matter most for predicting fraud. Baseline performance tracks how those features interact across different incident types, accident, medical, theft, fire, which means those relationships need to be well documented for the modelling to hold up.

Figure 18 Measure Baseline Performance

The Importance of Baselines and ML Flow in AI Models

Setting a baseline for an LLM or any AI model matters for judging its performance and success. It gives you a reference point to measure improvement or decline against, and helps stakeholders judge whether the model is actually meeting its objectives. Ongoing monitoring against that baseline is what surfaces trends, whether the model is getting better or worse over time. ML drift matters here too: it’s when the relationship between a model’s critical features shifts, signalling the environment has changed since the baseline was set. Understanding both baseline performance and drift together is what keeps model quality intact.

Feature drift is worth understanding on its own, especially given external shocks like COVID-19, which can shift the relationship between critical features significantly, sometimes pushing up previous-claims numbers with no wrongdoing involved at all. That’s why monitoring drift during deployment matters, to keep a clear read on how those feature relationships change over time. The speaker has experience monitoring ML drift in propensity models specifically, but it’s less clear how directly those principles carry over to LLMs; the foundational ideas probably overlap somewhat, but LLMs likely bring their own considerations when it comes to drift.

Building and deploying LLMs looks quite different from traditional machine learning. Rather than the gradual data distribution shift typical of ML, the priority tends to be integrating existing LLMs like Copilot or ChatGPT, using a Retrieval-Augmented Generation (RAG) interface to pull relevant data, claims policy information or student scores, say, from external databases rather than depending solely on a vector database. It’s really about making the most of existing technology to stay relevant as things keep changing fast.

Integrating student data into a chat system’s training model matters for better decision-making. Data quality and minimising LLM bias are the key considerations for getting the most out of it, and monitoring and evaluating the model’s capabilities, using metrics like J tech scores, is central to assessing how well it’s actually performing. It all comes back to careful data handling and rigorous evaluation being what makes a decision-support system genuinely robust.

Model Analysis and Quality Management in Machine Learning

Addressing model-related issues starts with root cause analysis and a look at the model’s quality: whether its complexity points to overfitting or underfitting, and whether the chosen algorithm even suits the data’s characteristics. It also means checking for model drift or concept decay, where features that used to matter stop being relevant, analysing for potential data bias, and verifying feature relationships against where the data actually came from. Managing the model’s lifecycle well, using something like CRISP-DM, is what lets models move smoothly into and out of production while staying interpretable and effective throughout.

Figure 19 Analyse Root Causes

Effective Use of AI Techniques in Fraud Detection and Prevention

Improving fraud detection and investigation skills is better served by focusing on the specific business requirement than trying to learn every machine learning technique out there. Start with what the fraud or anomaly detection need actually is, then look at the AI techniques that fit it, one-class SVM, isolation forest, autoencoders. Some of these need training first, an isolation forest has to be trained on anomaly detection data before it can make judgments, for instance. Going deep on a handful of techniques beats spreading thin across many.

Targeted AI models matter for fraud detection because they let businesses address specific needs instead of trying to absorb every corner of data science at once. Focusing on a clear business problem streamlines the whole approach to AI and produces more efficient, more impactful outcomes. A focused strategy like this improves the odds of successful fraud detection and makes for a better learning experience along the way too.

Figure 20 Improve System (Data, Model, Process)

Strategies for Improving AI Systems and Quality Management

There are a few strategies worth using to improve the system: retraining models to address drift, which often needs only a small amount of data and can be redeployed quickly. Analysing agent workflows and conversation traces is worth doing too, to check compliance with the intended process and understand what actually happened during interactions. Prioritising improvements by impact and effort matters, and so does cross-functional collaboration, moving away from records management, data governance, and AI system development operating in their own separate lanes.

Addressing quality failures and control issues properly takes team collaboration and root cause analysis on both fronts, evaluated against the current baseline, and that baseline might need re-establishing after retraining. Standardising model performance, data quality metrics, user feedback, and system health is what enables continuous monitoring. Statistical Process Control handles offline monitoring, assessing data quality independently of real-time interactions, while online quality management happens live, during an actual chatbot conversation, with results stored for offline review afterward.

Integrating online and offline quality reviews into the development cycle is what keeps performance at a high standard. A comprehensive total quality management rubric supports effective monitoring across the platform’s whole lifecycle. A pilot project is a good place to start, gathering real feedback on quality measures and showing executives that quality assurance is an ongoing commitment, not a one-off task. Regular reviews of the quality tools and figuring out how often to retrain the model help keep improving the process over time, and a sustained commitment to quality is really what drives better outcomes and a more robust platform.

AI Governance and Risk Management

A governance framework matters a great deal as organisations move into AI, particularly around data management. As teams build language models and AI agents, prioritising data governance is what prevents PII misuse and keeps users genuinely aware of what data they’re working with. The ethical side can’t be skipped either, addressing it properly is what makes AI use responsible and fair. Getting both governance and ethics right lets organisations get the benefit of AI without giving up individual rights or integrity.

Responsible governance is what keeps AI implementations safe and compliant, and that means carefully reviewing frameworks like the NIST AI Risk Management Framework (NIST AI-RMF) for guidance on managing AI-related risk. Categorising applications using the risk triangle from the EU AI Act matters too, particularly for steering clear of high-risk applications. A thorough risk management assessment is what aligns AI projects with broader business objectives while catching potential pitfalls early. Prioritising responsible practice this way supports innovation without losing sight of the wider societal interest.

The business is focused on managing and preventing fraud to protect revenue and keep payouts ethical. Avoiding fraudulent payouts is a clear goal, but there’s a real concern about how that affects customers, delays and extra friction in the claims process among them. Which is why balancing fraud management with a decent customer experience matters. The same balance shows up with AI in education: academic integrity has to be protected without letting any AI application undermine ethical standards or academic freedom.

Balancing ethics, bias, and academic freedom is central to keeping the integrity of the learning environment intact. That means treating both faculty and students as customers whose needs matter, without imposing rules so prescriptive they get in the way. One proposed solution for safer resource usage is a vetted list of approved materials, addressing the “shadow systems” problem where people bypass established protocols with unapproved methods instead. This is really about protecting institutional assets, data especially, while keeping the educational process responsible, and that’s what the future of academia depends on.

Educational institutions dealing with all this also have “shadow AI” tools like Copilot to contend with, which pose real challenges for data security and management. A lot of users gravitate toward these tools over the official ones, raising genuine concerns about data exposure and the need for solid controls. Building awareness of the risks that come with external tools is a key part of managing that change effectively. Organisations really need every information system accounted for, with processes structured to actually mitigate the risk.

Figure 21 Control and Standardise

Figure 22 Implementation and Next Steps

Figure 23 Continuous Monitoring and Feedback

Fraud Control and Accountability in Business Cases

Focusing on key outcomes in fraud cases matters a great deal, particularly around customer satisfaction, revenue loss, and fraudulent payouts. Controls need to meet regulatory requirements, HIPAA for medical claims, for instance, alongside frameworks like ISO 42000 and NIST. Evaluating a new business initiative means identifying the customer outcomes and benefits you want, but also every other area of regulatory compliance that might come into play. A balanced approach is what delivers both effective fraud prevention and stronger customer trust.

Sectors like higher education deal with the same challenge navigating HIPAA and GDPR, needing to comply with internal policy and external legal standards at once. Rubrics prove useful here too, helping stakeholders understand and communicate regulatory requirements more clearly. Running a conversation script through Copilot to check a regional rubric against HIPAA controls is one way to evaluate whether quality checks are adequate and spot compliance gaps, which builds regulatory awareness and stronger oversight in academic settings.

A structured approach to document management matters for accountability and effective record-keeping. Clear quality metrics for process definition, backed by regular audits, help confirm that designated people are actually meeting their responsibilities. Records retention needs addressing too, staying compliant with retention policies and deleting unnecessary data on time. All of this together improves operational efficiency while keeping the organisation aligned with regulatory guidelines.

Integrating audit checks into record-keeping matters when using AI and RAG systems for document storage, and every submission should go through thorough language and content validation. A submitted paper, for instance, should go into a private RAG database rather than a public one, so it can be checked the way Turnitin checks for plagiarism. Managing the content within the RAG so the original submitted document can be retrieved matters too, since that’s what allows feedback to actually go back to the person who submitted it.

Challenges and Solutions in AI Model Degradation

An early warning system for performance degradation matters for keeping AI models responsive to real-world change over time. Baseline checks on input data before deployment help catch potential drift or bias before it hurts performance. Different AI techniques carry different levels of confidence in their outputs too, which is why ongoing assessments like hallucination checks in LLMs matter, to evaluate how creative, or how off-track, the responses are getting. Regular anomaly detection, isolation forest methods among them, helps monitor model effectiveness on an ongoing basis. Staying proactive this way is what keeps AI systems reliable before users even notice a problem.

In Uganda, traditional credit scoring isn’t widely used, which has pushed the adoption of alternative approaches. One startup, for instance, has an LLM running in a local data centre to handle alternative credit scoring, analysing both traditional and non-traditional information to offer microloans to small businesses, expanding access to finance in the region.

Catastrophic events can throw off the reliability of that data quite a bit, though, a drought predicted two years out, say. Events like that cause data drift, where the statistical properties of the data shift and previously reliable metrics stop being trustworthy. It’s exactly why analysts need to adapt and reassess their methods when unforeseen circumstances hit, and understanding those implications is what keeps data analysis accurate and reliable over time.

Insurance Claims Process and Data Analysis

The insurance claims process relies on analysing previous claims data, and different claim types, accidents, fires, medical, have distinct mean values. Accidents, for example, have a mean of 3, fires a mean of 1, and medical claims a mean of 5. A catastrophic event can significantly alter that distribution, though, as the influx of new claims shifts the relationships between key features away from baseline expectations. That’s how you get increased means and claim amounts, COVID-19 being a good example of something that unexpectedly raised the average for certain claim types. Tracking those variations through the mean, standard deviation, and Z-score is what catches these shifts early.

Training a model well means a structured data split: 70% for training, 20% for testing, 10% held back for further production testing. Monitoring Z-scores per incident type helps track feature relationships, using a histogram as a way to visualise how the data structure is changing. Significant shifts there can be a signal that the model needs retraining. Colleagues with the right expertise, Tariq on credit scoring, for instance, are worth pulling into deeper conversations on this.

AI Techniques and Their Application in Data Analysis

Categorising AI techniques turns out to be genuinely complicated, since they can be classified by capability, environment, function, purpose, or specific methodology, deep learning and symbolic AI among them. No single taxonomy really covers every AI method; deep learning in particular underpins language models and frameworks like Transformers in a way that’s hard to fit into one clean category. Understanding how these techniques group together, and where the structural distinctions actually lie, matters for matching the right technique to the right task.

Monitoring for data drift in a deployed model, without waiting for some external catastrophic event to force the issue, starts with a solid baseline before deployment. That means systematically tracking the features that actually drive predictions, claim amount, incident type, claimant age, previous claims, for an insurance model. Scheduling regular assessments and automating alerts for shifts in the data or feature distributions is good practice too, so the model can get adjusted in time without someone having to catch it manually.

Before production, it’s worth taking a snapshot of how the Z-scores of features like claim amount, claim duration, and claimant age relate to each other, recording that as your baseline and recalculating it daily or weekly using the same data-splitting technique. Comparing current Z-scores against that baseline is how you catch a significant shift, checking the mean and deviations to see whether the underlying distribution, the bell curve, has actually changed over time.

The Z-score is a genuinely useful measure in data analysis, since it standardises data and helps calculate density, which is what identifies distribution peaks and outliers. Regular monitoring of these metrics matters, since daily or weekly fluctuations can signal a model needs retraining. No specific tools came up for tracking these changes, but explainability came up as a priority, with the suggestion that claims could be tagged for assessment to build understanding and transparency.

On explainability specifically, SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) are key tools for anomaly detection, helping determine whether a detected anomaly traces back to a known cause. If it can’t be explained, that’s often a sign something bigger has shifted in the data. Participants said they found the session’s insights genuinely useful for understanding anomalies and catching potential issues early, and the session as a whole went down well, with attendees thanking Howard for the material.