Unified Metamodel for Data Citizens

Executive Summary

The second week of unified meta-model discussions covered various challenges in data management education, including valuing working data over comprehensive documentation, data citizen education, and deliverables. It also worked through key concepts in data migration and management, understanding data latency requirements and how speed affects user experience. Low latency in data architecture, data steward responsibilities, and the challenges of handing over data stewardship responsibilities all came up too. Implementing data governance and organisational change was another major theme, with a proactive approach to data governance and training for effective data stewardship, covering how data stewards get involved in data governance and management projects, their role in business and data management, and the various roles on a data governance team. It closed on managing quality within a project, operationalising data stewardship, and assessing technology and managers’ involvement in decision-making.

Webinar Details

Title: Unified Metamodel for Data Citizens
Date: 15 September 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

Notes on Second Week of Unified Meta Model Discussions

Metadata’s importance came up early, since neglecting artefacts like data models and glossaries tends to cause technical debt and problems down the line when a system needs extending or growing. The week also covered core concepts and standardisation, with the current focus on the data citizen, a planned recap of community project progress, and discussion of project requirements and problems.

Figure 1 Data Management Unified Metamodel Deep-dive

Figure 2 Data Management Unified Metamodel Requirements

Challenges in Data Management Education

Assessing data management activities is genuinely hard without templates, scorecards, and benchmarks in place, and understanding every aspect, integration and definition included, matters a great deal. Educating data citizens on the fundamentals is necessary but difficult too, given how little experience most people have with the different artefacts, and connecting principles to activities brings its own challenges. Additionally, providing examples helps a lot here, and a lot of training participants actively look for them, since answering questions about how to use specific deliverables comes up constantly.

Maturity Assessments and Valuing Working Data over Comprehensive Documentation

The biggest challenge in a maturity assessment is finding who actually has the best example of what a good result looks like. Gabriel raised the agile manifesto’s emphasis on valuing working software over comprehensive documentation, and Howard pointed out that some development shops prioritise speed over thorough analysis as a result. The concept of “working data” came up in that same context, and defining it properly is still an open question.

Data Citizen Education and Deliverables

Properly defining and understanding data matters a great deal for making informed decisions with data sets, and rather than overwhelming data citizens with complex concepts, the focus should stay on teaching the fundamentals. Real progress has been made too, with individuals working on the Data Governance (DG), Data Quality (DQ), and Data Integration and Interoperability (DII) template libraries, and there are similarities between data access and data protection agreements worth discussing. DMBOK deliverables for DII can include defining data latency requirements to support business understanding.

Figure 3 Template Component Development

Figure 4 DII Template Library

Key Concepts in Data Migration and Management

Data migration takes establishing data transformation rules and acquiring operational metadata like data lineage and impact analysis, and red-to-green colour coding is a useful way to track progress. Examples and templates can demonstrate data exchange architecture and protocol, and metadata documentation, URI, and subject classification are all required for data services, while data access agreements set the terms for user access to specific data. Furthermore, data citizens need to understand the difference between data sharing and data access, and data latency matters a great deal here too, as a real aspect of data quality tied to timeliness.

Figure 5 DII DMBOK Artefact Progress

Figure 6 Data Exchange Architecture

Figure 7 Data Exchange Specification Artefact

Figure 8 Data Service Artefact

Figure 9 Data Sharing Vs. Data Access Agreements

Figure 10 Data Latency Requirements Description

Understanding Data Latency Requirements and the Impact of Speed on User Experience

Understanding data latency expectations matters a great deal in discussions and debates about data quality within organisations. Measuring data latency means defining how it’s actually measured and identifying the different areas where latency requirements exist, and a NASA case study is a good illustration of different areas and timings of data availability. Categorising the reasons for needing faster data, better forecasting or decision-making, for instance, matters too, and a Google assessment from 2012 found that a 400-millisecond delay led to a significant drop in search volume.

Figure 11 Timeliness

Figure 12 Data Latency Requirements Description

Figure 13 Data Latency Requirements Description pt.2

Figure 14 Data Latency Requirements Description pt.3

Figure 15 Data Latency Requirements Impact Assessment

Considerations for Data Delivery Speed and Accuracy

Speed matters a great deal for businesses when it comes to data delivery, Amazon has quantified that a delay of just 100 milliseconds can cost a 1% drop in sales. That’s why it’s worth asking business users specifically about their data needs, how frequently and quickly they actually need it. Getting data faster can come at the cost of accuracy, though, so working out an acceptable accuracy cutoff matters. Clients tend to prioritise fast data delivery without fully understanding the infrastructure and financial implications behind it, which is exactly why open discussions with users about their expectations and the associated costs are worth having.

Figure 16 Study on Data Latency Needs and Requirements – Molly Brown – Academia.edu

Low Latency in Data Architecture

A skilled data architect needs to question certain requirements and capture business needs properly to avoid scepticism down the line. Real-time data tracking matters a great deal in industries like mining, where human safety depends on immediate information, and operational business intelligence, monitoring product availability, for instance, can help optimise processes.

Evaluating data latency, analysis, decision-making, and the actions that follow matters for assessing the value and potential losses in operational business intelligence, and different types of data need different levels of consideration for latency and bandwidth. Autonomous driving and centralised control in mining are good examples of factors that influence data delays, and training data stewards using templates can benefit data citizens and support more efficient data management.

Figure 17 Data Latency Requirements: Operational BI

Figure 18 Data Latency & Bandwidth

Data Stewards’ Responsibilities and Need for Assistance

Data stewards need practical examples and training on creating and consuming artefacts in relation to their data strategy, business strategy, and business capability model, and understanding how to interpret and use these artefacts effectively matters a great deal.

Sustainable data management processes take a comprehensive tool and a proper reference architecture definition, since relying on Excel alone isn’t a viable long-term solution.

Day to day, data stewards manage operational data, make sure data production and consumption stay proper, and address quality issues as they arise. They also play a key role in defining data for data projects, supplying inputs, consuming outputs, and creating artefacts like data-sharing agreements and business glossaries.

Figure 19 Data Citizen Requirements

Figure 20 Data Stewardship Capabilities

Challenges and Strategies in Handing Over Data Stewardship Responsibilities

Data governance professionals often resist taking on data stewardship responsibilities, which is exactly why a smooth handover matters. “It’s not my problem” is a common pushback, and a resistance plan can help address that kind of response. Data stewards need to explain why they’re better suited to the role and demonstrate real benefits, improved business understanding and technical expertise among them, to build genuine engagement.

Figure 21 Data Stewardship Resistance Plan (OCM)

Implementing Data Governance and Organisational Change

Data governance is a genuinely complex process, getting the right people on board, sorting out the operating model, removing obstacles, and it takes a holistic approach that addresses all the interconnected elements together. Success depends on considering tangible events, rewards, performance management, skills, and ability, and creating awareness and desire through education and information matters a great deal. Providing artefacts and demonstrating how to implement data governance effectively helps too, and building a team to support and drive the implementation is what pulls it all together.

Proactive Approach to Data Governance and Handling of Responsibilities

After-action reports are worth using to understand what went wrong and revise procedures accordingly, which is what improves data governance over time. Assigning roles and responsibilities can be genuinely challenging when data stewards or data owners can’t fulfil them, though, and that’s often down to a lack of desire rather than ability. A multi-step approach, training, observation, then independent performance, helps overcome that, building both skills and confidence in meeting the responsibilities.

Figure 22 Data Stewardship Resistance Plan (OCM) pt.2

The Importance of Training Data Stewards for Effective Data Governance

Howard stressed how important proper training is in the data governance process, introducing the “flywheel” concept, the idea that incremental progress builds momentum and eventually success. Amazon’s own successful use of the flywheel, lowering costs, attracting more customers, increasing revenue, came up as an example.

Applied to data stewards, that means quick, efficient training matters a great deal for producing effective deliverables, creating a data access agreement, for instance. Artefacts delivered by well-trained data stewards reduce resistance and increase business ownership, which improves the organisation’s data management capabilities overall. The quicker data stewards become competent, in other words, the better those capabilities get, and the less resistance and more ownership follow.

Figure 23 What is the Data Steward Flywheel (Business & Technical)

Data Stewards in a Data Strategy

Data stewards play a key role in generating and maintaining a flywheel of data insights and learning, which drives improved business functions and more customers over time. That takes understanding a range of data management aspects: metadata management, valuation, modelling, data quality, parsing, matching, deduping, classifying, curating, cataloguing, and controlled vocabularies.

Training data stewards in understanding and modelling data in NoSQL and non-relational databases matters too, and building communities of practice where they can exchange tips and tricks on procedures, along with background templates to refer to, further strengthens their capabilities. Data stewards do face real challenges, though, interruptions in an office environment being a common one, which is why a robust process for quick learning and efficient deliverable production matters, letting data stewards genuinely take ownership of their capabilities.

Figure 24 What is the Data Steward Flywheel (Business & Technical) pt.2

Figure 25 Data Steward Fundamental Capabilities Training

Involving Data Stewards in Data Governance

Data governance aims to make data stewards more involved and responsible for their tasks, but plenty of stewards lack the necessary skills to do that well. Even experienced data modellers aren’t always proficient across every data evaluation aspect, and there’s a perception in some places that data modelling is no longer vital, which means modelling principles don’t always get followed. Data stewards still need to understand and provide input and receive outputs, though, especially in data modelling scenarios. Producing documents isn’t necessary, but genuine comprehension is.

Figure 26 Data Steward Deliverables

The Role of Data Stewards in Business and Data Management

Data stewards carry official roles and responsibilities for managing specific assets, conceptual data models included. They curate and manage those assets and are typically trained to read and understand models well enough to actually control and manage data systems. Additionally, Data stewardsThey should be able to define how a data system works even if they never produce a data model themselves, and while business people might work with data models too, that alone doesn’t make them data stewards. There can be multiple data stewards with different responsibilities, some working in cataloguing data, others in operational roles.

The Role of Data Stewards and Training in Data Management Projects

Data stewards play a key role across data management, covering areas like data modelling, profiling, curation, parsing, and matching. The DSBOK, introduced by the e-learning curve, goes into logical data models, perspectives, modelling purposes, and different reads and writes. A key question here is whether business-as-usual (BAU) and project-based data stewards should be working in parallel, and there’s a related discussion on the need for data management training templates once a project wraps up, so contributors have something shareable to work from afterward.

Figure 27 DSBOK Metadata Management

Roles of Data Governance Team

The data governance team handles operational data management and is made up of data stewards who monitor data capture and improve data quality. Some analyse BI reports and ensure data accuracy, while others define data and build data artefacts as part of the project, providing input, producing artefacts, and consuming artefacts along the way. They may move from project roles into operational ones, and part of that transition involves explaining the system they’ve built to the data stewards who’ll go on to consume it.

Managing Quality within a Project and Operationalising Data Stewardship

Data stewards play a key role in data management by defining critical data elements and creating quality rules, skills that carry well beyond the initial project into improving data definitions and managing quality expectations more broadly. Educating people on data management matters for providing input and delivering artefacts well, and case studies help with understanding use cases and how artefacts get created and consumed.

Data stewards can also contribute to defining an automation reference architecture and explaining the business value of reducing latency, and the technology in use should get assessed against other tools to understand its actual benefits. Involving managers in that assessment process matters too, for making genuinely informed decisions.

Figure 28 Data-value Driven Management Playbook

Figure 29 Revenue Generation: DIY CDMP Case-study

Figure 30 Data Management Automation