Data & AI Governance Unification – Final Episode

Executive Summary

This webinar presents key insights from the implementation of AI techniques across various sectors, with a particular focus on automating insurance claims and anomaly detection. Howard Diesel emphasises the central role of Data Quality management in successful AI deployment, detailing methodologies such as synthetic data generation and the application of anomaly detection in tools like Power BI. The webinar further explores advanced concepts, including the uncertainty matrix, Z-score for data standardisation, and the impact of the harmonic mean on model performance testing. Howard also addresses fundamental ethical considerations in automated decision-making and the balance between algorithmic efficiency and human oversight in financial fraud detection, pointing to the need for governance in AI applications within the health insurance sector.

Webinar Details

Title: Data & AI Governance Unification – Final Episode
Date: 18 September 2025
Presenter: Howard Diesel
Meetup Group: African Data Management Community
Write-up Author: Howard Diesel

Automating Insurance Claims and Anomaly Detection: A Case Study

Howard Diesel opened the webinar and shared that the focus would be on anomaly detection in insurance claims processing, specifically within the context of automating insurance claims. The business process outlined includes several key stages: first notice of loss, claim intake and validation, evidence processing, and fraud detection.

The objective is to develop a chatbot that assists claimants or brokers in submitting claims by guiding them through the process of providing necessary documentation, such as photographs, OCR recordings, police reports, and hospital information. Following the fraud detection and risk assessment phase, the process concludes with claim decisions, settlements, payments, and customer communication.

The interconnectedness of various frameworks within business processes matters a great deal in the insurance industry, as shown by a UML interaction diagram. Recent discussions focused on the challenges posed by chatbots, Large Language Models (LLMs), and data privacy, all key considerations in the industry’s day-to-day operations.

Central components of the business process, such as First Notice of Loss (FNOL), claim reporting, evidence gathering, risk scoring, fraud detection, and claim decision-making, show the complexity of these interactions. Aligning these processes with relevant Data Management knowledge areas also points to how much Data Governance and quality matter.

Effective management of customer, policy, and asset data through entity resolution matters for the system’s overall performance, reinforcing the ongoing need for Master Data Management (MDM) support even when using advanced technologies like chatbots. A framework that integrates these elements is key to improving insurance operations and reducing risk.

Figure 1 Unification of Data, Records and AI Governance

Figure 2 Case Studies

Figure 3 High-level Business Process

Figure 4 UML Interaction Diagram

Figure 5 Mapping to DMBoK Knowledge Areas

Figure 6 Reference and Master Data Management

Implementation of AI Techniques in Business

For implementing various AI techniques such as NLP chatbots, digital forms, identity proofing, OCR, computer vision, entity resolution, predictive modelling, and machine learning, teams need to understand the relevant frameworks associated with these technologies. Successful integration also requires attention to multiple factors, including Data Management, cybersecurity, corporate governance, ethics, and records management.

The interplay of these components shows the complexity of the AI landscape, pointing to the importance of a structured approach, such as the one provided by tools like Mario’s Prodago, which can align business drivers with practical use cases while managing background risks. Working carefully through potential challenges this way ensures that every important aspect is considered throughout the process.

In establishing a program that accommodates varying maturity levels, different challenges need to be addressed in a strategic and ordered manner. The process begins by defining clear objectives and use cases while considering relevant regulations such as ISO standards, corporate guidelines, and the recently introduced Saudi National AI Index.

Organisations often face overwhelming tasks, which can lead to confusion when working through compliance and governance frameworks. That’s why it matters to align business ambitions with a well-defined strategy, select pertinent use cases, assess constraints, evaluate capabilities, and adhere to AI governance requirements. This structured approach aims to simplify the effort and cut the risk of getting lost in compliance complexities.

Anomaly detection is a powerful tool in addressing Data Quality challenges, as it mirrors traditional Data Quality validation processes that rely on predefined business rules. When implementing anomaly detection, its value lies in identifying discrepancies across multiple Common Data Elements (CDEs) and features, rather than focusing solely on a single CDE.

This involves analysing relationships within the data to detect situations where expected changes have not occurred, prompting actions such as involving a business analyst to investigate further. This approach is similar to the historical practice in central banking, where analysts would verify unexpected stability in specific line items. Integrating AI techniques in anomaly detection enhances the ability to monitor data effectively and respond to potential issues efficiently.

Figure 7 Insurance Claims and Framework Mapping

Figure 8 Insurance Automated Claim

Figure 9 Insurance Automated Claim pt.2

Figure 10 AI Use Cases

Figure 11 Defining your AI Governance Requirements

Figure 12 AI Case Studies

Figure 13 Anomaly Detection Recipe

Figure 14 DQ Reasonableness Dimension

Anomaly Detection in Power BI

The Power BI tool developed for anomaly detection focuses on several key features, including claim amount, claim duration, claimant age, policy duration, and the number of previous claims. Within this framework, claim amounts can range from 0 to 100,000, while durations may extend up to 400 days, with an outlier of -200 days, and claimant ages can vary from 0 to 80 years.

Due to the significant differences in magnitude among these features, visualising them on a single graph can cause less significant features to be overshadowed. To address this, a technique is employed to compare these features effectively, without one dominating the other. The goal is to automatically identify anomalies, instances that deviate from expected patterns, while incorporating a human element to validate these findings.

Claimants face challenges when they urgently need funds for funeral coverage but are unable to receive payouts. A concern was raised about deferring the process to human checks, which could prolong the time taken and negatively impact the claimants. The importance of ensuring high Data Quality was emphasised, as poor data could lead to detrimental decisions in AI-driven processes.

Figure 15 Anomaly Detection

Figure 16 Insurance Claim Processing Application

Figure 17 Data Quality and Anomaly Detection Labelling

Creating Test Data for Anomaly Detection with AI

The process of anomaly detection requires a significant amount of test data, which poses a challenge for AI applications. To validate models effectively, a large dataset matters, as noted in the Wang dimensions of Data Quality. One solution to this issue is the generation of synthetic data, which involves analysing the patterns within existing production data. This method allows for the creation of datasets that can be used for testing without the difficulty of sourcing extensive real-world data.

The process of generating synthetic data involves using established patterns to create large datasets, which is beneficial for protecting Personal Identifiable Information (PII) by minimising exposure to sensitive data. While working with a central bank’s innovation hub, a data scientist demonstrated access to extensive production data across all departments, raising concerns about data sensitivity and security.

To ensure effective anomaly detection in machine learning models, teams need to generate synthetic data that not only reflects normal usage patterns but also incorporates deliberately introduced anomalies. By assessing whether the anomaly detection system can identify these anomalies, one can evaluate its effectiveness and address any potential shortcomings.

The process of identifying anomalies in clean data involves analysing claims that appear questionable and seeking validation from Subject Matter Experts (SMEs) or business personnel. This collaborative effort leads to a labelling process, where anomalies are categorised and explained, allowing for the development of a machine learning model to recognise these patterns. This approach plays a key role in improving Data Quality by teaching the model to identify and understand various types of anomalies effectively.

Data Quality Management and Its Importance in AI Implementation

Despite the growing demand for advanced Data Quality assessments, many organisations still prioritise basic completeness checks, which limit their ability to address more nuanced issues. An attendee shared her experience, pointing to the need for implementing reasonableness checks at the record or table level. Yet, she recognised that most companies are not yet prepared to make this shift.

The relationship between Data Quality and business drivers matters a great deal for organisational success. Understanding this connection lets businesses uncover valuable opportunities to identify and address Data Quality issues. Improving Data Quality matters for organisations that want to use artificial intelligence effectively and achieve their strategic goals.

Data Quality Management in Anomaly Detection

The City of Cape Town has an impressive data science team that uses anomaly detection for various processes. Recently, Howard engaged with a representative from the municipality, who invited him to conduct a workshop on Data Quality in AI during the Data Fest at the Windmill on October 3rd. In preparation, Howard shared that he planned to generate synthetic data, split it into training, testing, and production sets, normalise the data, and identify anomalies, followed by visual inspections. The final visualisation will present the analysis of different categories of incidents, providing valuable insights into the findings.

Anomaly detection can be effectively achieved through the use of three scoring models: SVM, Isolation Forest, and Autoencoder. This approach evaluates claims by assigning risk scores based on the number of models that identify anomalies. It grants 3 points for high risk when all three models detect an issue, 2 points for medium risk when two models do, and 1 point for low risk when only one model indicates a problem.

The accompanying visualisation clearly distinguishes between normal data and anomalies, with grey dots marking areas where no anomalies are detected. However, a significant challenge lies in elucidating the causes of these detected anomalies to Subject Matter Experts (SMEs). This method not only flags major outliers but also aids in classifying various risk levels, improving business analysis.

The dashboard’s functionality matters, as it allows users to visualise output and identify anomalies that require human review rather than automatic approval. The anomaly detection system flags potential issues well, but a human must still assess and verify these findings before making a judgment, ensuring claims are not unjustly denied.

Swift claim processing matters a great deal, especially in sensitive situations such as bereavement, to prevent unnecessary delays. Discovering false positives during the review process can also provide valuable insights for refining algorithms, helping them adapt to changing data conditions and cutting the risk of false negatives in future assessments.

Figure 18 Anomaly Review

Figure 19 Anomaly Review pt.2

The Concept of Uncertainty Matrix in Data Detection

The confusion matrix is a key tool for evaluating model performance, particularly in terms of precision and recall. Precision measures the model’s ability to detect true positives while minimising false positives, whereas recall assesses its effectiveness in identifying actual negatives.

A well-functioning confusion matrix facilitates the allocation of anomalies to human review, ensuring they are addressed promptly. The biggest challenge arises when the model fails to detect anomalies, leading to potential financial losses for the organisation. Proper labelling within this framework matters for effective analysis and interpretation of results.

In this analysis, synthetic data is generated with injected anomalies, including extreme claims, unusually long claim durations, implausible claimant ages, excessive policy amounts, and a history of high previous claims. The purpose of this approach is to evaluate the effectiveness of the data scientist’s model during training to detect these irregularities. If the model fails to identify the anomalies, it indicates a deficiency in sensitivity or other underlying challenges that need to be addressed. This serves as the foundational test in our evaluation process.

Figure 20 Anomaly Detection Recipe

Figure 21 Anomaly Detection Recipe pt.2

The Process of Injecting Anomaly into Power BI with Python Code

In our first test, we focus on injecting anomalies using Python code that is integrated into Power BI. The process involves running synthetic code to generate the necessary scripts. Within Power BI, I can edit the query to access the Python code, which serves as the data source. This code enables me to link directly to a dataset and use live data to generate synthetic data, demonstrating how we can effectively create and manipulate data within the platform.

Howard shared that he has generated a synthetic dataset with 10,000 records using a sample data set from production, which includes various incident categories. By analysing the mean, standard deviation, and shape of the existing data, Howard stated that it is possible to create a new set of normal data. This is because the generation process relies on an algorithm that uses statistics to produce random integers across different features. Each record is grouped by incident type, such as accidents or fires, ensuring that the shape of the data is specific to each incident type rather than uniform across the entire dataset.

The analysis of incident types reveals distinct shapes for different categories, such as accidents, fires, and medical incidents, each requiring careful consideration and attention. There are three main types of anomalies to monitor: point anomalies, which feature extreme claims or unusually long durations; collection anomalies, where groups of records align in a pattern but deviate from the overall trend, potentially confusing models; and contextual anomalies, which closely resemble the primary pattern yet differ slightly. Paying attention to these anomalies matters for accurate data interpretation and effective model performance.

In data preparation for machine learning, keeping a clean normal training set, free from anomalies, matters a great deal. To ensure this, the training data is typically divided into three subsets: 70% used for training, 20% for testing, and 10% for production, totalling 10,000 records (7,000 for training, 2,000 for testing, and 1,000 for production). Once the initial data is established, anomalies are introduced at the end of the process. To avoid clustering of these anomalies, the combined dataset is shuffled, ensuring a randomised distribution throughout, rather than having all anomalies grouped.

Figure 22 Generate Synthetic Data

Figure 23 Python Script for Synthetic Data

Figure 24 Inject Anomalies: For All Features

Figure 25 Generate Anomalies

Synthetic Data Generation and Anomaly Detection

The generation of synthetic data offers a valuable solution for efficient data handling, eliminating the need to extract large datasets from production environments. By creating synthetic datasets, businesses can quickly adapt to changes in procedures and variables, ensuring that their data remains relevant and accurate.

This adaptability matters because evolving business variables require the management and application of different data versions, particularly when addressing machine learning (ML) drift, where the structure of features may change over time. Staying updated on these changes allows organisations to maintain the integrity and usefulness of their data analysis efforts.

In developing your algorithm, incorporating a time-based feature by adding a date parameter to your feature list matters, and it simplifies the process without complicating the code. As you initiate the model, establish the shape of your features at training time and continually monitor for any significant changes that may occur. If the shape varies notably, it signals the need for retraining the model. Fortunately, certain routines allow for rapid retraining, enabling you to adapt to fluctuations and maintain model performance effectively.

To ensure effective model training and management, understanding the shape of the data and recognising when retraining is necessary due to significant changes matters a great deal. Continuous Data Quality (DQ) monitoring is employed to record initial metadata regarding data shape and to compare it with the current data constantly. When significant deviations are detected, alerts are raised, and retraining is called for. DQ stewards, who are closely in tune with business activities, play a key role in conveying changes to the model developers, keeping communication open to address any necessary adjustments or retraining.

The objective is to ensure that Subject Matter Experts (SMEs) are consistently involved in monitoring and understanding anomalies within the data. They must be equipped to explain the differences between normal and abnormal shapes, emphasising the importance of their vigilance in identifying and addressing changes.

Automated monitoring will conduct Data Quality (DQ) checks similar to those performed on reports, requiring regular refreshes and reviews. Currently, the data consists of 10,000 records, with 9,700 showing no anomalies, while 100 records have point anomalies, 100 have contextual anomalies, and another 100 have collection anomalies.

The process of anomaly detection involves a systematic approach in which Subject Matter Experts (SMEs) identify and label anomalies to train a machine learning model. As soon as an SME recognises an anomaly, they label it, providing the reason for the anomaly to facilitate the machine’s learning in detecting similar patterns in the future. This method involves injecting anomalies into the training data, enabling the system to recognise both specific anomalies and irregularities that occur within the data itself.

Accuracy metrics, such as precision and recall, are used to evaluate the model’s performance. Precision measures the proportion of flagged anomalies that are true anomalies, while recall assesses the model’s ability to identify actual anomalies. The F1 score, calculated as the harmonic mean of precision and recall, serves as a comprehensive metric of the model’s effectiveness.

Figure 26 Label and Combine

Figure 27 Measuring the Anomaly Detection Performance

The Harmonic Mean and Its Role in Model Performance Testing

The F1 score is a statistical measure known as the harmonic mean, which provides insight into a model’s performance by balancing precision and recall. Unlike the arithmetic mean, which high values can skew, the harmonic mean emphasises lower values. If recall is significantly low, the F1 score will reflect poor model performance, as seen in an example where precision is one and recall is 0.1, resulting in an F1 score of 0.18. This characteristic of the F1 score serves as an important indicator that action, such as retraining the model, may be necessary to improve its effectiveness.

F1 refers to a metric that uses harmonic means to measure predictive performance, showing variances from the average through different calculation methods. Regularly testing algorithms matters, to make sure they have accurate predictive power rather than relying on random guessing.

A high true positive rate indicates effective predictive capability. In contrast, a low rate suggests the model is ineffective and generates unnecessary workload for humans who must verify the results, leading to a high false positive rate. To improve efficiency, minimising human involvement in validating outcomes and improving the algorithm’s accuracy matters a great deal.

Figure 28 F1-Score: Harmonic Mean

Figure 29 ROC Curve

Confusion Matrices and Anomaly Detection

The pursuit of a perfect classifier in anomaly detection involves the aspiration to create a model that can accurately identify every anomaly without any errors. Although achieving absolute perfection is nearly unattainable for most algorithms, evaluating how closely a given algorithm approaches this ideal standard still matters.

A key tool in this evaluation process is the confusion matrix, which distinctly categorises actual normal instances against predicted outcomes, both normal and anomalous. Using the confusion matrix, we can monitor discrepancies between predicted and actual classifications, which shows the frequency of true positives, false positives, true negatives, and false negatives. This analysis provides valuable insights into the effectiveness of the algorithm in detecting anomalies, guiding further improvements in model development.

The primary objective is to address the issue of false negatives in fraud detection, as many anomalies are currently being overlooked. This oversight poses a significant risk, suggesting that the existing fraud detection models may not be effectively identifying fraudulent claims. As a result, a proposal has been made to slow down claim approvals, allowing personnel to manually review claims more thoroughly. The intention is to enhance detection efforts, particularly when automated systems fail to detect potential fraud adequately.

Figure 30 Confusion Matrix

The Implementation of SVM in Power BI

In a recent analysis using a one-class Support Vector Machine (SVM) model, I shared my Python code implementation within Power BI, showcasing the results, including the ROC curve. The model’s performance was concerning, as it produced a low ROC curve, indicating its effectiveness was akin to random guessing, which was a surprising revelation after initially perceiving the model as performing well. This experience pointed to the importance of thoroughly evaluating model metrics to avoid overconfidence in seemingly promising results.

The ROC curve illustrates the performance of a model analysing normal data without anomalies, suggesting that while it identifies a baseline of what is considered normal, it also detects low-level anomalies. In an ideal scenario with no anomalies present, the ROC curve would appear as a flat line at the bottom, indicating no detections. This counterintuitive behaviour of the graph, which dips downward rather than showing a typical upward trend, presented a challenge in understanding the model’s anomaly detection capabilities.

Figure 31 ROC Curve in Power BI

Anomaly Detection in Data with Python Code

In the Python code, the focus is on analysing normal data by excluding anomalies to generate a Receiver Operating Characteristic (ROC) curve. The training uses a one-class Support Vector Machine (SVM) with a decision function that assesses the distance of data points. However, this approach reveals that some normal records are incorrectly flagged as anomalies, raising questions about the integrity of the data. While the complexity of the topic may pose challenges for some, participants are engaged and appreciative of the insights being shared, even without immediate feedback.

Anomaly detection routines are surprisingly simple to implement. Typically, these routines can be coded in around 50 lines of clear and understandable code, which shows how anomalies are introduced and how key metrics, such as the ROC (Receiver Operating Characteristic) curve and confusion matrix, are calculated.

During the discussion, an attendee emphasised that for effective model performance, the ROC curve should ideally be positioned in the top left corner; however, the analysis conducted revealed no anomalies, as the dataset comprised entirely of normal data. The team’s dedication to sourcing and structuring the appropriate data samples from production was key to ensuring the accuracy of their results.

The process of generating synthetic data revealed challenges within the dataset, which, when disregarding possible anomalies, resulted in a high number of false positives. This situation calls for a reassessment of what counts as a normal training dataset; if the dataset is cluttered with anomalies, it misguides the model into learning that bad claims are good. That makes it important to collaborate with Subject Matter Experts (SMEs) to understand the outliers identified by the model and ensure accurate definitions of good claims, improving overall Data Quality and model performance.

Figure 32 One-Class SVM and ROC Curve

Figure 33 Why is the False Positive Rate so Low?

Data Anomaly Detection and Ethical Considerations in Claims Processing

In the context of data analysis using Power BI, a significant challenge is thoroughly understanding the quality of the data to identify anomalies effectively. Analysts often face difficulties in discerning why certain data points, such as claim amount, duration, days, age, policy, and previous claims, are flagged as anomalies by their systems.

To address this issue, a proposal has been developed to create an agent that collaborates with Subject Matter Experts (SMEs) to investigate the causes of these anomalies. Advancements in language modelling technologies, such as LLMs, could also enhance the analysis of language used in claim submissions, providing deeper insights into identifying irregularities in claims.

Large Language Models (LLMs) use specific structures and vocabulary relevant to particular contexts, such as car accidents or cancer treatments. When users employ inaccurate language, LLMs may identify discrepancies, raising concerns about the validity of the data. An example of this challenge emerged in a project involving anomaly detection, where the team struggled to verify flagged data fields because they relied on algorithms.

This situation pointed to an ethical dilemma: justifying flagged inconsistencies without confirming actual errors. Unreasonable data needs to be approached with caution, encouraging engagement with Subject Matter Experts (SMEs) to investigate anomalies through methods like reviewing recordings and photographs, classifying these as false positives when no issues are found.

The process of refining a detection model involves revisiting the labelled data to identify acceptable patterns versus anomalies, which can be labour-intensive, particularly in verifying the accuracy of various claims. This approach enables a more focused analysis, concentrating on genuine anomalies rather than examining all data, resulting in a more efficient use of resources.

Effective communication about the system’s intent, clarifying that not all flagged items are incorrect, can help create a more comfortable investigative environment. This parallels issues faced by law enforcement, where preconceived notions can skew perceptions and escalate situations unnecessarily, emphasising the importance of context and clarity in data interpretation and communication.

Figure 34 Power BI Demo: Generate Synthetic Data

Role of Algorithms and Human Intervention in Financial Fraud

An attendee raised a concern about relying on algorithms to detect patterns in fraudulent transactions, pointing to the need for human intervention at key points. They recounted a personal experience in which, after being defrauded, the bank contacted them about suspicious transactions but ultimately placed the burden of verification on them.

This approach was cost-effective for the bank, as investigating every anomaly would have been more expensive than reimbursing customers for unauthorised transactions. While this method may expedite resolution, it raises concerns about honesty in claims, particularly when individuals might be tempted to exaggerate or misrepresent circumstances to recover lost funds.

Balancing Ethics and Governance in AI Applications in Health Insurance

The implementation of AI in the health insurance sector presents significant complexities, especially in claims processing and patient information management. One of the key challenges lies in the use of prescriptive AI to analyse health claims, which raises ethical concerns related to security and Data Governance.

The dual focus on fraud detection and responsible handling of sensitive patient data points to the need for rigorous standards and frameworks to ensure the ethical use of AI models. Balancing the use of AI for improved health outcomes against safeguarding patient information matters for the future of the industry.

Feature selection and engineering play a key role in Data Governance, particularly in the context of responsible AI practices. Howard points to the need to use data as intended by patients to prevent potential privacy violations, and stresses the importance of balancing the business value of AI applications with the value returned to customers, a principle outlined in the EU AI Act. Additionally, Howard noted that a legal expert had emphasised the importance of thoroughly evaluating feature choices in model training to ensure compliance with and adherence to ethical standards. The objective is to enhance the customer experience by speeding up payouts, enabling customers to receive their funds as quickly as possible.

The fundamental aim of our approach is to ensure that AI systems are consistently aligned with their intended purposes. A significant concern arises from the potential misuse of AI, particularly in sensitive situations such as those involving individuals with serious health issues like cancer. For instance, if an AI fraud detection system incorrectly flags legitimate claims, it can lead to distressing and damaging interactions with these vulnerable individuals, who may already be facing severe challenges. This scenario shows how much strong corporate governance and AI oversight matter in preventing such incidents and protecting both customers and the company’s reputation.

To effectively balance data privacy and the use of AI models, establishing appropriate controls and consistently asking relevant questions matters. The focus should be on using algorithms to enhance the quality assessment of data and detect anomalies, which simplifies the overall process. This approach facilitates a deeper understanding of how to use AI responsibly while ensuring that privacy considerations are prioritised.

Data Use and Anomaly Detection in Automated Decision-Making

There is a need for clear customer consent regarding the use of data, particularly in scenarios involving automated decision-making and training. It emphasises the importance of distinguishing between a customer’s right to deny automated decisions based on trust issues and the company’s ability to retain Non-Personally Identifiable Information (PII) and other transactional data even if PII is removed. Howard also pointed out the challenge of re-identification (re-ID) when analysing data and the necessity of addressing anomalies detected during model development, especially for data collected years prior.

When dealing with Personally Identifiable Information (PII), data must be properly anonymised to prevent re-identification. As a data architect, one must address concerns about the mathematical feasibility of re-identifying individuals from aggregated data. The fundamental principle is that as long as the information remains aggregated and individual identities cannot be discerned, it can be used in models. However, in small populations, such as a town with only 123 residents, it becomes increasingly challenging, as specific age bins or brackets may allow for re-identification. The crux of the issue lies in demonstrating that data cannot be re-identified mathematically.

The harmonic mean is valuable for addressing skewed data, which often leads to the disregard of the conventional mean. Howard expressed appreciation for the harmonic mean’s unique approach to data analysis, noting how it provides a different perspective. An attendee also recalled a lecturer’s wisdom, emphasising that a mean without a standard deviation lacks significance, which reinforces the importance of incorporating measures of variability in statistical analysis. The conversation pointed to the importance of the harmonic mean in comprehensively understanding data.

Understanding the Power of Z-Score in Data Standardisation

The Z-score is a statistical measure used to understand the position of a data point relative to the mean of a dataset, calculated as the difference between the actual value and the mean, divided by the standard deviation of the dataset. This process involves splitting the data, coding the split, validating it, and normalising the claims. A Z-score with an absolute value greater than 3 indicates that the data point deviates significantly from the normal range, suggesting a potential anomaly or challenge within the dataset.

The Z-score is a powerful statistical tool that transforms data points into a standardised range between -5 and 5, significantly reducing the impact of outliers. By applying the Z-score, one can effectively compare variables such as claim amount, claim duration, and claimant age, as it enhances the differences between these metrics. This standardisation matters a great deal when dealing with vast datasets, as it allows for clearer insights by ensuring consistency in data representation. For optimal training of models, using Z-scores rather than relying on the original data values matters.