Data Quality Framework & Methodologies for Data Professionals

Executive Summary

This webinar makes the case for how central data elements are to data quality, covering approaches to defining data elements and dimensions, data quality expectations, reasonableness, and ensuring data quality within applications. It works through data movement and analysis, understanding root causes and building improvement plans, data quality tool architecture, organising and documenting data for analysis, development approaches for data quality, effective planning with minimal changes across improvement cycles, data quality in agile development, and clears up some confusion around entity associations along the way.

Webinar Details

Title: Data Quality Framework & Methodologies for Data Professionals
Date: 21 July 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

The Relevance of Shoe Size in a Master Data System

Howard Diesel used the example of “shoe size” in a master data system to get the discussion going, treating it as reference data. Drew, one of the webinar attendees, proposed that “shoe size” should be treated as reference data rather than customer data, and Howard agreed that while “shoe size” falls under a domain, it may not need to sit inside a customer master data system at all. Drew’s reasoning was that customer data usually consists of demographics like name, address, and age, while “shoe size” can vary across different lines of business. Howard also shared an anecdote about a shoe salesman who struggled to gauge customer demand and run surveys, a good illustration of how hard buying decisions can be to get right.

Figure 1 Critical Data Elements (CDE) Criteria, Sourced from Data Governance for Certification

Challenges of Maintaining Accurate Shoe Size in Master Data System

Sticking with the “shoe size” analogy, limited availability of certain sizes can be a real challenge when it comes to selling all products, and the accuracy of reference data can be affected by things like currency fluctuations and rate changes too. Because the “shoe size” attribute was so volatile, it ended up getting removed from the master data system entirely, with a forecasting algorithm used to estimate it instead of relying on real-time updates. Monitoring attribute changes matters a lot for data quality here, and interestingly, older individuals tend to see longer intervals between “shoe size” changes. Understanding critical data elements (CDEs) is essential for assessing data quality, and the SIPOC procedure, sources of information, calculation process, output, and recipients, can help with that. Data plays a real role in business decision-making too, which is why having a decision inventory matters for understanding the components of the decision process.

Understanding the Importance of Critical Data Elements in Data Quality

Doing SIPOC properly depends on understanding data quality clearly, and a critical data element (CDE) is what guides the selection process and keeps unnecessary information out. Prioritising the most important CDE should come down to business use cases derived from the data strategy.

As a starting point, the CDEs people complain about most are usually the most critical ones. Prioritising them means evaluating their business value and organisational relevance, and the MoSCoW approach (Must, Should, Could, Won’t) is a useful way to work out the importance of each CDE.

Figure 2 Choosing Critical Data Elements (CDE) and Definition Flowchart

Approaches to Defining Data Elements and Dimensions

In Power BI, having a row or dataset lets you work with multiple values within a dimension. Defining a data element means specifying a minimum and maximum value and writing a rule description that helps identify attribute failures, a contract number, for example.

Starting with a use case means identifying the critical data elements first, then the consumers of the data, then the right dataset. If you’re building a data product instead, it’s the reverse: start with the dataset and work out the critical data elements for each consumer from there.

Giving context to critical data elements means listing dimensions like the status of the measure, along with the rule status and message.

Figure 3 Data Quality (DQ) Expectation Template

Data Quality Expectations and Minimum/Maximum Values

Defining critical data elements means establishing a set of data quality rules and evaluating the data against them. It also means figuring out the best system of record for those elements and identifying the authoritative system for creating, updating, and reading the data. Getting accuracy right takes considering business processes too, particularly attributes used across multiple processes.

Having a minimum value matters for prioritising relevant information in data analysis, since without one, you end up with confusion and wasted effort. Both minimum and maximum values matter in sales planning cycles, and it’s worth noting the maximum won’t always be 100%, that varies by scenario. An SPC diagram can help identify exceptional cases above the maximum and below the minimum, and it’s worth paying attention to those cases and striving for completeness, even where 100% isn’t realistically achievable.

Reasonableness and Data Quality

When analysing gender population, the recommendation is to start around 60%, moving gradually to 70% and then 80% over time. Reasonableness checks matter for catching excessive growth, a 300% increase, say, and reasonableness itself should generally sit between 15% and 20% to catch outliers. Most dimensions work fine with just minimum values, though some need maximum values too, to keep ratings in check. Logical, consistent attribute values are the goal, but reasonableness can also cap increases, a 15% cap on balance sheet line items, for instance.

Historical analysis helps determine what counts as reasonable growth, and if a reasonableness error crops up, it’s worth investigating, though it doesn’t automatically mean the data itself is inaccurate. Data quality professionals play a genuinely important role in keeping data both accurate and reasonable.

Importance of Setting Data Quality Expectations and the Role of a Data Architect

Implementing a data quality framework starts with setting expectations and genuinely listening to what business professionals are concerned about. Automation, ongoing monitoring, and daily maintenance are the key objectives, and data professionals, the data architect especially, manage the data throughout its lifecycle and are central to the framework’s architecture.

That architecture covers managing application challenges, ETL processes, developing data products, and reporting analytics, and identifying and resolving data quality issues comes down to SLA checks and root cause analysis. The data architect plays a key role in building a comprehensive diagram and lineage to spot issues and get a holistic view of the system, and root cause issues typically turn up at the application level, in system and entity relationship design.

To sum that up: before rolling out a data quality framework, it pays to set expectations early and genuinely take on board what business professionals are worried about. Automation, ongoing monitoring, and daily maintenance should sit at the centre of the framework, and it falls mainly to data professionals, particularly the data architect, to manage the data across its lifecycle and keep the framework’s architecture sound. That covers application challenges, building data products, and reporting analytics through ETL, with SLA checks and root cause analysis as the way to keep data quality on track. The data architect is well placed to build a comprehensive lineage diagram to catch and resolve issues, giving the whole system a more holistic view, and again, most root cause issues tend to surface at the application level, in system and entity relationship design.

Figure 4 Managing the Data throughout its lifecycle

Ensuring Data Quality in the Application

Getting data accurate and reliable takes attention at several points throughout the data management process. It starts with assessing how much protection the data model actually provides against incorrect data being created in the first place, and avoiding duplicate records during data entry matters too. Data validity is worth verifying during ingestion, to keep things accurate as data moves from applications into the warehouse, and any reporting errors are worth analysing to trace back the root causes, consistency, timeliness, or latency issues among them. Challenges around data moving from applications to the warehouse or data mart need identifying and addressing, and latency itself, the time data takes to move from entry to reporting, is worth measuring and monitoring. Root causes for specific data quality challenges are worth chasing down too, and setting clear quality requirements from the start is what stops insufficient data flowing through the system unchecked.

Figure 5 Common Root Causes of Poor Data Quality

Data Movement and Analysis

Understanding how data moves matters a great deal for identifying problem areas, and dimensions help make sense of potential issues in those areas. The Ishikawa diagram is a genuinely useful tool for identifying what to investigate, people, training, executives, materials, and it’s worth considering data quality damage at various stages, accuracy included, and running root cause analysis to trace issues back to their source, poor system design or inadequate enterprise database design, for instance. Accuracy is a key concept when it comes to real-world applications and agreed sources, and reference data plays a vital role in ensuring accuracy by mapping and verifying information. Data analysis itself remains a genuinely valuable tool for getting relevant, accurate information.

Figure 6 DQ Dimensions are applicable at the different stages of the data lineage from source to the BI report

Understanding Root Causes and Implementing Improvement Plans

Improving application quality means analysing various areas: application design, database design, data integration. The Ishikawa diagram is a genuinely helpful tool for identifying root causes here, and it can be customised with process, application, database, and policy categories. Once root causes are identified, categorising them and building an improvement plan around specific issues, usability, accessibility, completeness, timeliness, is what actually solves problems and helps prioritise improvement efforts effectively.

Data Quality Tool Architecture

Maintaining high-quality data matters a great deal when integrating various systems, and that applies across the enterprise, operational records, master data, applications, all of it.

Auditing and reconciliation both demand solid data quality, which is worth keeping in mind when choosing a data quality tool. The ideal tool offers a metadata repository for data quality rules and trend analysis results, execution services for validating data, automating cleaning, and running audits, and support for data stewardship services to identify and fix data issues.

Data quality tends to break down into three dimensions: machine programs that can detect and fix issues, machines that detect while people fix, and people who both detect and fix. Analytics and a range of different databases are needed to keep metadata, assessment results, and reporting up to date, and probes are needed to analyse databases like JSON, Oracle, and SQL.

Data profiling, data quality assessment, understanding data models and processes, and monitoring and reporting, all of that forms the core of a data quality tool architecture.

Organising and documenting data for analysis and data quality

A knowledge repository makes it much easier to understand the processes and data flows within the database, the tables, views, and attributes involved. The data dictionary helps define data dimensions and metrics across different databases and organisational units, and tracking data sharing and application processes means documenting data lineage and external sources in the data flow itself. A well-documented framework, covering data models, application design, and business processes, is what supports proper analysis and state reconstruction. The star schema design for data quality brings together data groups, metrics, measures, data quality dimensions, processes, and organisational units, and a genuinely important part of data architecture is building the team itself, which starts with establishing a clear “way of work”.

Development Approach for Data Quality

Howard weighed up the best approach for addressing data quality issues, comparing Waterfall and Agile development methods, and made the case that Waterfall might actually be more effective for building a reliable foundation for data quality, since it prioritises accuracy from the outset.

Agile brings flexibility for changing requirements, sure, but it may not be the best fit when you’re facing numerous data quality problems at once. Howard’s suggestion is a Waterfall strategy for the initial iteration, to establish boundaries and metrics, with a more Agile approach coming in afterward for ongoing monitoring.

Identifying the root cause of data quality issues and resolving them before pushing toward the desired quality level matters a great deal here. Ben, one of the participants, agreed that a Waterfall approach makes sense initially, followed by a more Agile approach for later iterations, and Raj backed using Waterfall to lay the groundwork too, since that groundwork simply can’t be rushed.

Howard’s overall view is that starting with Waterfall is genuinely important for success, particularly for building a strong foundation for data quality, with a switch to Agile worth considering once the initial objectives and end goal have actually been reached. His preference, in the end, leans toward staying with Waterfall and prioritising that strong foundation.

Figure 7 Data Quality Improvement Lifecycle “Plan-Do-Check-Act”

The Importance of Effective Planning and Minimal Changes in Improvement Cycles

Fully understanding and addressing the underlying causes of problems matters a great deal in the improvement cycle, and keeping changes to a minimum during the planning, fixing, and checking stages is what avoids unexpected issues cropping up.

Agile methodology can encourage innovation and allow for change, but limiting how many changes get made during improvement cycles tends to work best. Howard pointed to the Pareto principle as a useful way to identify and prioritise the handful of issues driving 80% of the problems.

During improvement cycles, it falls to an architect to define the work process and make sure it gets executed properly. Howard acknowledged the case for using agile methodology in future improvements, but still stressed how important a Waterfall approach is for reaching the next target level effectively and keeping planned fixes on track.

Data Quality in Agile Development

Boosting product quality from 60% to 70% depends on assessing and understanding the situation correctly from the start, since addressing one issue can create others if it’s not handled carefully. Prioritising which issues to tackle in each sprint or iteration matters a great deal, and some methodologies, Agile included, aren’t necessarily suited to every aspect, data quality and distributed ledger projects among them. The DMBOK covers data modelling notation for representing associations, and the Merise notation represents associative entities specifically, while Peter Chen’s ‘Chen notation’ uses a diamond shape to link attributes. Veronica asked about the Chen notation specifically, wanting to understand that particular data modelling notation.

Clearing up confusion on entity associations

Chen’s notation uses a diamond shape to depict entity associations, with attributes spread out rather than packed into the entity box itself.

Figure 8 Clearing up confusion on entity associations