Data Warehousing, BI, Big Data & Data Science for Data-Driven Executives

Executive Summary

This webinar covers today’s competitive landscape and why effective Data Management matters so much for business success, particularly in the context of artificial intelligence (AI) and data science. Howard Diesel explores that transition and the critical role robust data flow, quality, and governance play in AI model evaluation and deployment.

As organisations face challenges around Data Quality and digitalisation, integrating business knowledge with data insight becomes a genuine necessity. Trustworthy data underpins AI adoption, which is why a strategic approach that redefines Data Management as a core business discipline matters so much. The webinar closes by stressing that collaboration between technology and people is what fosters an environment where Data Management genuinely contributes to sustainable organisational success.

Webinar Details

Title: Data Warehousing, BI, Big Data & Data Science for Data-Driven Executives
Date: 06 February 2025
Presenter: Howard Diesel
Meetup Group: African Data Management Community
Write-up Author: Howard Diesel

Data Management in Business and the Role of AI

Howard Diesel opened the webinar by sharing his strong belief in AI’s potential, its usefulness in daily tasks, and the value of various large language models. He also pointed to a significant challenge, though: convincing businesses of how critical Data Management is as a discipline in its own right.

Figure 1 Data Warehousing, BI, Big Data & Data Science

Figure 2 Data Management = Most Important Business Discipline

The Shift from AI to Data Science

Francesco Puppini presented the previous webinar Howard mentioned earlier, talking through the critical need for effective Data Management practices as businesses adopt AI and data science technologies. A key point was the “garbage in, garbage out” principle, which captures the consequences poor Data Quality has on AI outcomes.

Plenty of organisations invest heavily in AI without establishing proper data frameworks first, which leads to inconsistencies and trust issues with the results. The concept of “shifting left” came up too, arguing for early investment in data infrastructure and understanding data collection processes to minimise costs and improve efficiency. Successful case studies help bolster the argument for proactive Data Management, showing its foundational role in driving organisational success and saving money in the long run.

Data Management and Data Flow in AI

Howard moved on to the value of leaning on thought leaders in the AI field, pointing to Andrew Ng’s campaign for data-centric AI. Ng’s argument is that many practitioners prioritise model development over Data Management, which can cause real problems during model evaluation and deployment. Howard also referenced a Google-developed concept called “data cascades,” illustrating how problems in data collection, labelling, analysis, and cleansing can cascade into later stages of AI development. Addressing data-related issues early on prevents challenges further down the line during model work, which is really the case for prioritising Data Management in AI projects from the start.

Plenty of challenges, especially in descriptive analytics, come from poorly defined issues, declining sales, revenue drops, decreased customer engagement, which is exactly why a clear problem statement matters so much in data analytics. A clear problem statement also points to something bigger: effective Data Management extends beyond IT into business-driven practices, particularly data stewardship and governance. Proper management of reference and Master Data matters a great deal too, and it should be led by the business rather than IT.

Figure 3 Data Cascades in High-Stakes AI

Figure 4 Forbes Article on Andrew Ng

Figure 5 Challenges with AI & Data Cascades

Figure 6 Data Work and Model work present in the Data Cascade Diagram

The Challenges of Data Quality in AI Model Evaluation and Deployment

Google’s work points to real challenges in AI model evaluation and deployment, particularly the risk of models getting abandoned post-deployment. The distinction between Business Intelligence (BI) and machine learning (ML) can stretch out the deployment process, since ML needs low error rates and correct decision-making confirmed, unlike BI’s more straightforward reliance on user acceptance testing of data products.

The Data Cascade’s extended timeline can drive up costs and effort, particularly when models need revisiting or discarding altogether. Howard noted that Andrew Ng stresses how much the effectiveness and responsibility of AI decisions depend on high-quality, accurate, complete data sets, and reiterated how poor Data Quality can lead to genuinely disastrous outcomes. Data, in that sense, is the essential “food” for AI, fuelling training and decision-making capabilities, with both accuracy and completeness standing out as vital dimensions of Data Quality.

Howard also spoke to how critical Data Quality and governance are for accurately reflecting the real world, pointing to concerns about data reliability, especially data acquired from external sources or not regularly updated. That can introduce real bias, focusing predominantly on high-net-worth individuals while neglecting other ethnic groups, for instance.

A lack of completeness creates real challenges and affects how useful AI actually is, since it depends so heavily on the quality of its underlying data. The key takeaway: unless data accuracy and comprehensiveness get prioritised, the effort risks being futile and time wasted. Stakeholders need to recognise the necessity of addressing these data issues to actually improve decision-making.

Figure 7 “Conclusion”

The Challenges and Insights in Data Management and Digitalization

A common issue in Data Management projects is the gap in understanding between data engineers and the domain experts behind the data sources. Howard recounted a situation where a €5 million digitisation project used 19 disparate data sources, which posed real challenges for effective Data Management. Suggestions to consolidate the data for better efficiency met resistance too, stemming from the differing perspectives of those creating data systems versus those managing data infrastructure.

Having roles like data architects to build a cohesive enterprise Data Model matters a great deal, and there’s a broader need for data scientists to understand data complexities more fully. Data scientists often focus on quickly loading data into models without recognising the underlying integration challenges involved.

The Intersection of Data and Business Knowledge

The critical intersection between data knowledge and business understanding shows up in engineering teams that feel overwhelmed from lacking access to business insight. Despite real technical skill, these teams struggle to connect data initiatives with business objectives, often overlooking how important conceptual and logical modelling is for understanding and communicating business needs.

Howard pointed out that many professionals only recognise the initial stages of the data life cycle, neglecting the planning and design phases that effective Data Management actually needs. That gap points to why modelling matters for gaining deep insight into a business: if a model doesn’t make sense in plain language, it’s unlikely to make sense in data terms either.

The Importance of Data Management in AI Development

Recent observations from Google Engineering point to the critical, often undervalued role data plays in AI development. Howard noted these observations found 92% of machine learning models encountering data issues, many of them avoidable with effective Data Management, reflecting a broader trend in the conference landscape where presentations lacking “AI” in their title struggle to get accepted.

Howard shared that AI developers spend 80% of their time on data preparation, a task better suited to data engineers, which raises the question of whether data engineers are being excluded from the process, or providing data that isn’t optimally structured for AI applications in the first place. He also referenced Francesco Puppini’s webinar again, which stresses the importance of the “last mile” of business intelligence (BI), the transition from the data warehouse to the final BI model, as Francesco put it. That phase, he suggested, is genuinely critical for delivering value from analytics.

Figure 8 “Observations from Google”

Figure 9 Extract from Forbes Article

Data Management in AI

Data Management’s critical role for AI comes down to four key areas: Data Quality, Metadata Management, Master Data, and Data Governance. Data Quality means identifying and resolving issues through automated rules and cleaning processes, while Metadata Management covers labelling, tagging, classification, and lineage. For Master Data, identifying and automating reference data sources matters a great deal.

Howard also pointed to ontologies as a way to standardise data across various sources, particularly in clinical trials, which has supported reliable data integration. Data Governance matters here too, for managing risks and bias and ensuring data gets handled responsibly, ultimately pulling these elements together into a streamlined approach to Data Management that supports automation and efficiency.

Figure 10 Intelligent Data Management

The Role of Technology and People in Data Management

Intelligent Data Management marks a shift away from traditional, manual human resources for Data Governance, largely because the sheer volume of data can’t be managed efficiently without the right technology behind it. AI and automation genuinely boost productivity and data classification, but people and policies still matter a great deal in this process. Technology alone isn’t enough; effective Data Management needs human involvement and established processes alongside it to ensure accuracy and compliance. A collaborative approach that brings technology and human expertise together is really what makes Data Governance successful.

The Evolution of Data Management in Business

Howard reflected on past challenges in establishing effective Data Management governance, noting difficulties in gaining business ownership and commitment. Some divisions have maintained good programs, but there’s a growing need for data literacy training and skills development in tools like Power BI and Tableau, since data has become genuinely essential for business success.

Integrating Data Management principles into everyday tasks without overwhelming employees matters a great deal, and Howard drew a parallel with technology like GIS, which people often use unconsciously. He also discussed the DCAM framework, distinguishing Data Governance from oversight, and cautioned against the perception that Data Governance is overly burdensome, stressing the need to clarify its benefits to actually encourage adoption.

The Need for Trustworthy Data in AI Adoption and Management

AI is a genuinely powerful business discipline, reshaping operational value through continuous learning and innovation. To maximise its effectiveness, though, responsible decision-making depends on high-quality, unbiased data, often called trustworthy data, which means a real shift in Data Management practices toward delivering reliable data products tailored to business needs.

As organisations bring large language models (LLMs) into their operations, they need to make sure those models are populated with their own specific data and insight. Effective Data Management becomes critical here, since simply using the latest data isn’t enough; a structured approach is what’s needed to actually manufacture trustworthy data. Data Management, in the end, goes well beyond technology capability, covering planning, design, and consolidation efforts to ensure Data Quality for AI models to succeed.

One attendee stressed how important Data Quality is in business, using the analogy of retailers refusing to put subpar meat or vegetables on their shelves, to illustrate the disconnect in accepting flawed fraud data instead. They also raised concerns about the long history of failed AI and machine learning initiatives, clarifying that the issue usually isn’t the models themselves but the quality of data feeding into them. The goal is identifying and addressing the root causes of AI failure to actually improve performance, rather than dismissing AI altogether.

Figure 11 Data Management = Most Important Business Discipline

Redefining Data Management as a Business Discipline

Elevating Data Management to a recognised business discipline means demonstrating its sustainable business value through quantifiable metrics within Data Management programs. The real challenge lies in engaging stakeholders, particularly in AI and business intelligence, to actually recognise and appreciate the role data plays in driving economic value and innovation.

Starting with the right data lets organisations use data science to identify opportunities for business innovation, building a continuous cycle of value creation. Data executives need to reframe the conversation around Data Management, stressing its critical contribution to informed decision-making and overall business success.

Figure 12 6+1 Attributes of Data Management Discipline

Figure 13 Value Creation “Flywheel”

Figure 14 Data Executive: Call to Action

The Importance of Data Management and Business Knowingness in Organizational Success

Howard stressed how important a clear Data Management framework is for distinguishing data responsibilities from IT, letting business stakeholders engage effectively with Data Management. He also pointed to the need for business users to take ownership of Data Quality and relevance, keeping things aligned with business needs.

One example Howard shared from past experience was in sell-side research, showing how business-driven decisions on data sources and transformations led to better outcomes, with IT aligning to support those initiatives. He closed by noting that a challenge from a business executive underscored the need for data professionals to build up their business and financial acumen, to communicate and collaborate more effectively.