Unified Metamodel for Data Professionals

Executive Summary

This webinar covers key topics in data management, including why unified metamodels, data governance, and metadata management matter, and how Power BI fits into data strategy. Howard Diesel and Paul Bolton make the case for practice courses in achieving successful results, and work through the challenges of training data stewards. The webinar also touches on data governance and metadata in implementing a data warehouse, the role of quality and reporting in data management, the need for oversight and assurance in data management processes, and knowledge management and data literacy in the oil and gas industry.

Webinar Details

Title: Unified Metamodel for Data Professionals
Date: 22 September 2023
Presenter: Howard Diesel and Paul Bolton
Meetup Group: Data Professionals
Write-up Author: Howard Diesel

Introduction to Unified Metamodel

The webinar centres on the Unified Metamodel, a unified metadata framework. Howard has been working on it for three weeks, with Paul handling data governance templates and Veronica working on data integration templates. Using a specific example, Howard explained the need for a single source of truth for metadata, since industry standards typically keep separate meta-models for different tools, which isn’t enough for real integration. He then demonstrated how they link metadata components, a business glossary, data model, ETL, data quality, to bring everything together.

Figure 1 Unified Metamodel

Figure 2 Metadata Management: 5W1H

Unified Method and Metamodels in Data Management

Starting from the business glossary matters when searching for reports, data quality profiles, configuration, or database management systems. John O’Gorman’s suggestion is to use a Knowledge Graph to bring all relevant information together and support data sharing. Effective data management also means showing both horizontal lineage (from business glossary through conceptual models, logical models, design patterns) and vertical lineage (data stores, data quality, data inspiration, data warehousing). Meta models matter here too, and while they vary from person to person, they’re an integral part of a unified method in data management.

Power BI and Data Management Specialists

Power BI lets users create PBIT files, read data from the Power BI model, and generate JSON files for documentation. The platform aims to establish lineage and connect different data elements with the business glossary definition sitting at the centre as the reference point, and graph technology can link business glossary terms, logical data model attributes, physical data model fields, code sets, applications, and systems. That’s exactly why data management specialists need to define procedures, principles, and policies around data governance and quality, and why adopting good practices and understanding data capabilities matters so much for succeeding in the role.

Figure 3 Conceptual Metamodel

Figure 4 Example Metadata Repository Metamodel

Figure 5 Data Management Specialist

Figure 6 Data Management Specialist Requirements

Ensuring Oversight and Assurance in Data Management Processes

Howard clarified the difference between documenting data management processes and knowledge management, noting that a lot of people confuse metadata management with master data management. Understanding the role data governance plays in oversight and assurance matters here, and effective data management needs real controls in place, not starting a procedure without defined data models, for instance, and having the right inputs before work even begins. Prioritise those factors, and data management ends up genuinely optimised for better results.

Figure 7 Oversight & Assurance

Defining Procedures and Processes for Data Governance

Defining a process means identifying input, proactive, and concurrent controls, and a data steward can help define a business definition or cluster to build a data model around. Oversight matters for making sure the right policies, decisions, and processes are in place and actually achieving the results expected, and implementing data quality means defining frameworks like Plan-Do-Check-Act or TQM.

Principles, policies, procedures, reference architecture, guidelines, tools, and techniques all need establishing within data governance, and a business glossary paired with a data dictionary is what lets users navigate through associated elements. Where business clusters aren’t available, data models can be used to create business glossary definitions instead.

Figure 8 Oversight (Watch Care)

Figure 9 Assurance 3: Line of Defence (3LOD)

Figure 10 Business Start from the Glossary

Unified Metamodel and Data Catalogue

A data catalogue with metadata comes together by combining a glossary, a data dictionary, and a data model. Building a template project takes a data capture template, a procedure for populating it, and a scoring system to evaluate the job, and the starting point for any data management artefact is an ID and a business key. The “SIPOC” covers the artefact’s supplier, input, procedure, output, and consumer, and a scorecard assesses the quality and value of the artefact against categories like process execution time, cycle time, efficiency, output quality, and value to the consumer, which is what procedures ultimately get evaluated against.

Figure 11 DM Project Template Starting Point

Figure 12 DM Project Template Starting Point

Figure 13 Design DM Artefact

Importance of Practice Course in Achieving Successful Results

Creating a successful template, a default artefact, default SIPOC, and default scorecard, in the right sequence of artefact, procedure, and scoreboard, matters a great deal. A business case study works well as a training example for creating these artefacts, and the DIY course includes a leaderboard to encourage practice and track progress. That said, completion on a specific quiz in the course is low, only 6 out of 37 students finished it, which is exactly why encouraging more time on the practice course matters for actually succeeding in the real exam.

Figure 14 CDMP DIY Leaderboard Use-case

Analysis and Presentation of Data Using Power BI

Howard analysed the data and presented the results on a leaderboard, stressing how important data governance is and voicing concern about the lack of focus on data security and warehousing. His recommendation was more attention on certain practice quizzes to encourage better habits, and he shared metadata too, data dictionaries and quality scores among it, demonstrating how to navigate the data and identify student information. When asked about a model or illustration of how the data elements relate to each other, Howard said it’s currently being developed and will get folded into Power BI.

Figure 15 CDMP DIY Leaderboard Use-case

Figure 16 Official CDMP DM Fundamentals Results

Figure 17 CDMP DIY Leaderboard Metadata Demo

Figure 18 Knowledge Area Analysis

Figure 19 Metadata & Data Quality Dashboard

Data Strategy and Power BI Integration

Howard hasn’t transitioned to a data model yet, given the current data strategy template already links various IT systems, use case categories, reference data elements, digital strategy, business strategic pillar, employees, and issues. Starting from the DMBOK has helped scope the project, though, and they’ve identified 152 templates to produce, including 30 custom ones not defined in the DMBOK itself. Howard’s Power BI loads in the background, visually representing the data, status, KPIs, knowledge areas, deliverables, data models, reference data lists, metadata repositories, DMBOK tables, and naming conventions in the table designs make it easy to extract and analyse specific elements in Power BI, as one appropriately named table demonstrated.

Figure 20 Data Management – Use-case Portfolio

Figure 21 DMBOK Deliverable

Figure 22 Metadata Repository Dashboard

Options for Metadata Management in Commercial Software

Informatica stands out as a top commercial software choice for metadata management, thanks to its connectivity and use of AI for active metadata creation, though the cost can be prohibitive for small and medium enterprises in Venezuela. Open data sources like Data Hub can automate data collection, but caution is necessary to prevent irrelevant data overload.

Software Development and Data Visualization

Howard also discussed the challenges of getting a CFO’s signature given compliance with BCBS 239, stressing how important it is to manage foreign data and visualise it properly. The metadata repository itself is a Power BI tool, not a spreadsheet, and Howard plans to use John’s Q6 knowledge graph to build a more adaptable data repository and improve visualisation further.

Power API is currently the tool used to connect and extract data, and the need for generic templates and procedures across organisations came up too, since customers want quick delivery on these. Howard closed by presenting artefacts being developed for DII, including missing information on data lineage, impact analysis, data transformation, data migration, and data latency requirements.

Figure 23 DII Template Library

Data Governance and Data Latency in Project Development

The project centres on defining its architecture and templates, which involves translating an architecture into an Excel spreadsheet. It also aims to create a data mapping and exchange pecification and define the data service and access agreement. Additionally, the team has to keep compliance with data privacy regulations like GDPR front of mind, protecting personal information carefully, and also address data latency, by defining and assessing it as part of the project. A spreadsheet tracks progress and connects the different elements, and Mr. Bolton, a contributor, expressed real enthusiasm for his involvement in the project.

Figure 24 Data Exchange Architecture

Figure 25 Data Access Agreement Description

Figure 26 Data Sharing Agreement Template

Updating and Aligning Project Plans in Power BI

Howard stressed using project plans to stay on track and accountable for daily or bi-daily outputs, suggesting Power BI for updating the status and format of those plans. Templates for a data strategy came up too, along with the importance of aligning terminology and reference data.

Test data and swipes were added to the project alongside an action plan and roadmap, and an architect raised concerns about the connection between uplift objectives and the business strategy, which prompted further work. Howard’s closing point was how important it is to align project work with business priorities in the strategy document.

Figure 27 Data Governance Data Strategy

Review of Operating Model and Data Governance

An operating model review is needed given the changes to the business glossary, and there’s a discussion still needed on the data maturity assessment template. A gap in the scoping column could affect Power BI, and a communication plan with a recipe for action and measurement is currently underway.

Work is also progressing on knowledge area principles and the use case portfolio templates, and the data people gap is being analysed, with a focus on domains and required staff. Resourcing options and channels for finding people are being explored too, and a catalogue and scoring method for policy documents is being developed. Comments and questions on the progress are always welcome.

Figure 28 Data Governance Business Glossary

Figure 29 Data Governance Communication Plan

Figure 30 Data Governance Communication & Training SIPOC

Figure 31 Data Governance Data People Gaps

Figure 32 Policy Scorecard

Clarifying the Unified Model and Requesting Examples of Artifacts

Howard stressed the importance of a comprehensive list of inputs and dependency documents that go beyond deliverables alone. Paul shared that he had struggled with connecting different knowledge areas and understanding how to create artefacts without clear examples, but was still excited about using a Knowledge Graph to connect all the dots and eager to start putting everything together. Veronica then asked for guidance on creating artefacts, given the lack of examples in the book, and a participant sought clarification on what “unified” actually means in the Unified Model.

The Importance of the Unified Metamodel in Data Modelling

A unified meta-model matters a great deal for connecting various meta-models and avoiding siloed work, though building one gets compared to building a death star, daunting and nearly impossible. Informatica has connected its own tools to the data lineage and data dictionary elements, which shows real progress toward a unified model. The unified metamodel matters for assessing data management maturity and accessing the evidence needed to accurately score artefacts.

Challenges in Training Data Stewards and Educating People on Data Management

Organisations run into real challenges from a lack of critical data elements, policies, and scorecards. Building policies and procedures for every practice within the organisation takes defining everything that’s been created and learned, and policies and procedures need creating for every specification and piece of mission evidence.

Training data stewards is hindered by the absence of examples and a clear understanding of data quality expectations and data access agreements, and educating people on the DMBOK and data management theory often lacks the practical examples needed for real comprehension. Misunderstanding data latency and why it matters can lead to confusion and even loss of personnel, which is exactly why tangible, relatable examples matter so much for people applying this in their own roles.

Knowledge Management and Data Literacy in the Oil and Gas Industry

In industries with high turnover, Knowledge Management matters a great deal for preventing the loss of expertise. Leaky Knowledge Management, losing knowledge when experts leave without proper documentation or practices, is becoming a real issue in organisations, and a defined process and record of Knowledge Management is what avoids that.

Data literacy and metadata management matter a lot in the oil and gas industry specifically, and outsourcing projects within the industry can call for a different data and metadata management approach. Managing metadata and maintaining databases remains an active topic of discussion within the sector.

Implementation of Data Governance and Metadata Programs

Data governance means assessing the business’s level of capability and identifying where a metadata program is actually needed, and it’s responsible for establishing and resourcing that program to manage the organisation’s metadata. A Data Management Maturity Assessment (DMMA) helps identify high-risk areas, the absence of a data quality program or metadata program, for instance, and a business case gets developed and approved to justify the need for these programs before they start.

Establishing the metadata program means hiring data professionals with the right knowledge, skills, and experience, and data governance is what keeps the company compliant by identifying weaknesses through DMMA and taking corrective action from there. Once it’s established, data professionals take on executing and completing the work.

Data Governance and Metadata in Implementing a Data Warehouse

Shifting organisational culture to prioritise data is genuinely challenging. An organisation might have a data strategy and governance structure in place already, but it’s not always fully developed, and a properly established function responsible for governing data and ensuring its quality still matters. The focus for the coming year should be implementing a data warehouse and developing a metadata strategy, and the Elite tool can help build conceptual models and business glossaries along the way.

Data lineage and trust in data quality matter enormously, and foundational metadata and data quality need to be in place before anything else can really advance. A concurrent focus on establishing both the data warehouse and quality makes sense, and setting quality expectations while explaining data quality clearly to stakeholders is what actually builds understanding.

Importance of Quality and Reporting in Data Management

Effective data management means addressing the quality of current data usage and layering in additional elements for improvement. Integrating metadata, niche data, and notifications can genuinely improve data quality and management, and templates and examples matter for guiding people through the reporting frameworks they need, ones that help identify business requirements, accountability, data sources, data refresh rates, and cost management.

Without templates and guidance, people can end up frustrated and overloaded, which is where practical courses and sessions help, giving a more hands-on approach to learning data management. Asking questions and learning from them matters throughout, since that’s what actually builds understanding and drives improvement.