Data Quality Framework & Methodologies for Data Executives

Executive Summary

This webinar works through the key strategies and considerations for understanding and improving data quality. Howard Diesel makes the case for data literacy and critical data elements in writing, and for accurately assessing data quality and cost impact. He explores the benefits of data science, cloud computing, and natural language processing, along with the challenges and ethics of AI in decision-making. The webinar also covers the importance of a common language, data governance, and building data teams.

Webinar Details

Title: Data Quality Frameworks & Methodologies for Data Executive
Date: 28 July 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

Strategies for Understanding and Improving Data Quality

Addressing data quality issues matters a great deal for executives, and techniques like force field analysis and common language assessment can help pin down where an organisation actually stands on data quality. Challenges here often trace back to people and culture rather than technology, which is why the ADKAR approach and stakeholder awareness sessions matter. Evaluating decision-making capabilities and understanding who’s actually on the data quality team matters too, and communicating the benefits of data quality improvements to stakeholders is what gets their commitment and investment. Comprehensive training programs for data literacy round things out, for everyone involved in data quality initiatives.

Figure 1 Critical Business Case Questions

Importance of Data Literacy and Data Quality in Writing

Data literacy shapes people’s confidence in their own writing quite a bit, and a lack of it can feed into negative self-perception. Changing the data quality culture really comes down to demonstrating success and winning projects one at a time to build participation. Evaluating the return on investment for data quality programs matters too, with cost reduction typically being the primary benefit rather than revenue generation.

Addressing data quality issues means prioritising challenges and making well-informed decisions, and mapping out a cost taxonomy helps make sense of the cost and impact involved. Management consulting firms can add value here too, running data quality reviews that improve writing outcomes.

The Importance of Accurately Assessing Data Quality and Cost Impact

Howard walked through the key points of a report on data quality assessment, which found that exaggerating the costs of data quality issues can actually hurt the credibility of the initial assessment. He also pointed to a recommended approach: selecting the last 100 records of a dataset for review. The report highlighted just how frustrating poor data quality can be too, citing an example of someone spending hours trying to fix data for a report.

The executive team itself was fairly split on where the problems originate, seven out of eight believed data quality was a major issue, while one attributed the problems to how businesspeople use the data. Howard also mentioned Tom Redman’s “Friday afternoon measurement” approach, selecting a critical BI report or a frequently complained-about data asset as the basis for assessing data quality.

Figure 2 Friday Afternoon Measurement (Thursday Morning – Saudi)

Importance of Critical Data Elements and Cost of Defective Data

Analysing critical data elements means working out which key attributes actually need focus in a table or report. The colour of clothes being purchased is one example of a critical data element worth analysing, and a review of 100 records was conducted to identify challenges with it, categorising records by attributes like size, colour, and amount. Interestingly, no errors turned up in the “name” attribute, but 33 of the 100 records had errors elsewhere, roughly 30% of the dataset.

To put a number on the impact of that defective data, the rule of 10 was applied, which holds that it costs ten times more to process defective data than perfect data. Which means processing 100 units of perfect data would actually cost less than processing those 33 defective records alone. Communicating that metric to business owners is worth doing, since it drives home the real consequences of working with flawed data. Rather than chasing perfection outright, the suggestion is to set improvement targets instead, reducing the cost from its current level down to 160, or whatever specific target makes sense.

Figure 3 Applying the “Rule of Ten”

Data quality assessment and scope definition

Howard’s advice is to assess data quality for critical data elements before applying any model, starting with BI reports or data extracts that users are already complaining about rather than diving straight into the application databases. Tracing data lineage back to the source matters, as does getting an initial figure for the business case to build support for change.

It’s worth weighing the organisation’s actual capacity for change too, and identifying the challenges and restraining forces at play. Additionally, Howard recommends avoiding long projects with no immediate results, since those tend to breed murmuring and concern.

Analysis of SWOT Analysis and Impact of Data Science

SWOT analysis is a genuinely useful tool for looking at an organisation’s Strengths, Weaknesses, Opportunities, and Threats, and it should focus on identifying the internal challenges an organisation is facing alongside its areas of strength and good process. Opportunities and success stories in the market are worth considering too, as are threats like industry failures, regulations, and faster-growing competitors.

Data and technology can shape the analysis further, but restraining forces, scepticism and fear stemming from high-profile failures, for instance, need addressing head-on. The level of impact for each force can be scored on a scale of one to five, and driving forces, success stories from companies like Facebook, Apple, Amazon, Netflix, Google, and Spotify, can be used to build strategy and awareness.

Figure 4 Change Assessment Force-field Analysis

Figure 5 Example: Business Impact of Data Science

Benefits of Data Science and Cloud Computing, Utilisation of Ishikawa Diagrams, and Strengths of Chat GPT

Adopting cloud computing and getting access to good data scientists has driven real growth in benefits, and data quality methods have played a genuinely important role in addressing data issues. The force field model has proven very effective for achieving goals, strengthening driving forces while weakening restraining ones, and turning restraining forces into driving forces has been shown to cut down on missed opportunities.

The Ishikawa diagram helps visualise challenges, which then supports more effective communication, and asset management challenges tend to touch customer experience, organisational agility, competitive advantage, productivity, cost, and risk all at once. Small wins in resolving issues add up to real overall improvement. ChatGPT’s natural language processing capabilities make it a strong contender in the chatbot market, and its effectiveness and potential really show up in video processing and related applications.

Figure 6 Issues Caused by Lack of Leadership

Figure 7 SWOT Analysis

The Power of Natural Language Processing (NLP) in Video Analysis and Document Summarisation

Combining technology with natural language processing (NLP) has made it possible to extract and summarise video content quickly and in real detail, and NLP’s ability to add chapters makes analysing videos granularly a lot easier. It can be customised across different organisations too, and gets more effective as the dataset grows. ChatGPT is an efficient, scalable tool for a range of tasks, and conversing with AI, or using tools like Grammarly, is both fast and genuinely useful.

The number of NLP applications keeps growing, and the pace of progress is impressive, Google recently released its own bot, for instance. That said, contextual awareness is a real weakness in video analysis, since NLP only picks up audio, not visuals, and it’s worth staying mindful of other weaknesses too, bias and dependence on OpenAI among them. The speaker worked out they have roughly 211 videos on hand, totalling somewhere around 200-300 hours.

Figure 8 Change Assessment Force-field Analysis (FFA) for ChatGPT

Analysing the Challenges and Ethics of AI in Decision-Making

Organisations weighing up Bing or Bard for AI use need to consider whether they’re on Azure first. OpenAI’s web scraping has raised its own ethical concerns and potential legal action, and technical limitations in AI’s accuracy can lead to hallucinations or fabrications when it can’t find specific information.

One case study showed AI-generated legal information errors leading to a genuine court disaster. Calculating the FFA score means assessing opportunities, strengths, threats, and weaknesses together, and Excel’s scoring mechanism assigns values to analyse internal factors more precisely, adding two for strengths, subtracting two for weaknesses, adding one for opportunities, and subtracting one for threats.

Figure 9 Scoring the FFA (Template)

Analysing the Benefits and Ethical Concerns of Chat GPT

Using ChatGPT on videos was rated positively overall, though there are ethical concerns around using people’s names in videos, which needs careful editing to protect privacy, and contextual awareness matters a lot for avoiding exposing sensitive information unintentionally. Potential bias isn’t a major concern for the intended use case here, but the dependency on OpenAI is still being evaluated, with other vendors under consideration too. Numerical analysis matters for impact assessment and future measurement, since SWOT analysis on its own doesn’t really quantify anything.

Figure 10 FFA: ChatGPT Modelware Systems

Common Language and Data Quality

SWOT analysis is genuinely useful for prioritising strategies and decision-making, while data maturity assessment helps introduce new capabilities like data science. Establishing a common language matters a great deal for data quality, since it improves communication and helps ensure data accuracy, and critical data elements need to be well-defined, which is where business glossaries and definitions come in. An article by David Hay, written with Zachman, called “Developing and Adopting a Common Language”, is a genuinely good resource on this.

Figure 11 Establish a Common Language Approach

Data Governance and Building Data Teams

As Ronald Ross, Mark Atkins, and Terry Smith all made clear, data governance and risk management matter a great deal for organisations. Language groups are being set up to preserve and manage different languages, and identifying which regular employees are actually proficient in working with data matters for analysis. Data provocateurs, meanwhile, challenge the status quo and push for better definitions and practices in data management, and regular employees like Andrew need to actively participate in the data generation process too.

Data quality (DQ) is central to effective data teams, which cover entrepreneurs, developers, defenders, data quality leaders, data trainers, and data transformers. Understanding how data-driven the organisation actually is matters for building an informed business case, and the Continuum model can assess that progress, ranging from data resistance through to knowledge-driven.

Figure 12 The Work Team

Figure 13 DQ & Data Team “Superhero’s”

Figure 14 Trails of Great Data Decision-Makers

Strategies for Driving Data-Driven Decisions and Enhancing Data Quality

Positioning for growth and overcoming stakeholder resistance takes drawing on diverse data and viewpoints to make genuinely informed decisions. Making sure decision-making rests on high-quality data means seeking out supplementary data and assessing data usage habits and consistency through scoring techniques. Executives and data professionals both need to be involved in reviewing and challenging data-driven decisions. Data literacy matters a great deal, though it does take real cost and time to train for, and beyond basic statistical analysis, forecasting and graph interpretation are genuinely valuable data abilities to have.

Levels of Data Literacy and Applying Data in the Workplace

Data literacy, managing data included, is a genuinely important skill in today’s world, and a lack of awareness, understanding, or knowledge can lead to costly mistakes, which is why it’s worth quantifying the cost of ignorance. Data citizenship is the first level of data literacy, learning to follow data policies, followed by improving data quality itself. The final level is applying data analysis and forecasting in your own work, which is what lets individuals genuinely contribute to their organisation’s growth and success.

Figure 15 Data Quality Training

Figure 16 Important Data Quality Business Skills & Outcomes

Data Quality and Decision-Making in Business

Howard worked through the different levels of quality dimensions in data and analytical models, and why quality checks at each level matter for improving decision-making. Training is necessary, he noted, to understand the size of the data quality problem, the root cause, the level of ignorance involved, cost analysis, force field analysis, and language. Howard also touched on the current state of business decision-making based on data, the need for culture change and awareness, and running decision maturity interviews with executives to set a roadmap for improvement.

Weaknesses in data quality and in decision-making tend to correlate, and addressing them in the business strategy means ongoing monitoring and review of data quality challenges across the organisation. Howard’s closing advice: executives should build a business case to address these issues directly.

Figure 17 Critical Business Case Questions

Canadian Institute for Health Information – Providing Open Data and Analysing Health Information

The Canadian Institute for Health Information (CIHI) collects health information from government and hospitals, packaging it as time series and advanced analytics, then sells or publishes it to provide insight into health-related topics like alcoholism, drug use, and pregnancy. CIHI analyses the data to produce insights and refers to the output as “information” rather than “data.” Additionally, they have their own data quality expectations too, and will return data to the Ministry or hospitals if it doesn’t meet those standards, focusing on data quality dimensions like fitness for purpose, fitness for use, relevance, and timeliness.

Figure 18 CIHI Information Quality Framework

Figure 19 CIHI’s Quality Dimensions

Figure 20 Quality Principles

Timeliness and Quality in Data Publishing

Data accuracy and trustworthiness matter a great deal for reliable, informed decisions, but validating and assessing received data can be genuinely challenging, which is why quality review and continuous monitoring matter so much. Chasing perfection in data collection can actually leave you with no data at all, so balancing different needs while staying focused on clarity and accuracy is what actually produces trustworthy information. Shared responsibility and ownership are key principles in data management, and assurance needs to run throughout the information life cycle rather than just get checked at the end, with continuous monitoring and filtering keeping data quality intact and preventing drift.

Figure 21 Quality Principles

Key Steps and Framework in Data Quality Management

Juran and Deming built a framework for managing data quality around a step-by-step approach that ensures accuracy and reliability. The cycle runs through creating, capturing, and submitting data, then processing, analysis, and dissemination, and CIHI provides guidelines for the last three steps specifically to keep them appropriate.

The framework covers evaluating and documenting metadata and data, running quality assessments, and identifying improvements, with the underlying goal being more knowledge, trust, usage, and overall data quality. CIHI’s own dimensions line up with existing approaches for the most part, though some differences do exist.

Figure 22 CIHI’s Information Quality Framework

Figure 23 CIHI’s Quality Activities

Data Quality Assessment for Multiple Organisations

Howard worked through key aspects of data quality assessment, empirical contradictions, uncertainty, inadmissible precision, standardised vocabulary, stressing how important it is to build dimensions that actually make sense to the business. He shared a list of R functions used in data quality assessment too, tying them to concepts like missingness, and touched on producing reports like heat maps of missingness for variables such as waist circumference and body height. He also pointed to additional resources like Excel spreadsheets and addressed a question about data quality assessment across multiple organisations.

Figure 24 Data Quality > Consistency > Range and Volume Violations / Contradictions

Collection and Assessment of Health Data by Government Ministries

The Ministry of Health mandates that everyone involved in healthcare, individuals and organisations alike, gather and report data, which then gets assessed for quality. Data that falls short of the standard can get excluded from the overall dataset, though valid conclusions can still be drawn as long as the majority of contributors are providing accurate data.

Other sectors, globally significant banks among them, run similar assessment systems, and Howard noted that in Saudi Arabia, data catalogues are being developed alongside data owners being assigned to improve data quality. Canada’s Ministry of Health, for its part, established CIHI as the country’s health information body.

Enforcing data quality using data catalogues

Some ministries set up separate companies to sidestep confidentiality issues and conflicts of interest when monetising collected data. Data catalogues matter a great deal for effective data management here, giving a way to enforce data quality, categorise data assets by governance and curation, and visualise and disseminate information about those assets.

A data catalogue is really a metadata repository, collecting data lineage, models, and business glossaries, and letting data assets get separated into different categories for better management and understanding. Integrating data catalogues matters, though it comes with its own challenges, data scientists still wanting access to uncurated data being a common one.