The Conceptual Modelling Journey Map for Data Professionals

Executive Summary

This webinar explores the intricate world of Data Integration and Data Modelling. This instalment of ‘the Conceptual Modelling Journey Map’ covers the role and importance of concepts, literacy, and validation processes in economic analysis. It addresses the challenges and potential of concept-based programming, different modelling languages, database schema, and various types of Data Models and database implementations.

Howard Diesel shares on the complexities and relationships in UML, Entity Relationship Modelling, dimensional modelling, focal point modelling, database normalisation, fact-oriented modelling, semantic modelling, and data storage. The integration of Data Models in various domains, data architecture, and collaborative practices for concept modelling are also highlighted. The webinar concludes with an exploration of advanced analytics, graph databases, wide data sets, document databases, and the evolution of digital services.

Webinar Details

Title: The Conceptual Modelling Journey Map for Data Professionals
Date: 22 August 2024
Presenter: Howard Diesel
Meetup Group: African Data Management Community
Write-up Author: Howard Diesel

Role of Concepts in Data Integration and Integration

Howard Diesel opens the webinar and begins on the use of canonical hypergraphs as part of a model for integrating data. He shares ‘the Handbook of Conceptual Modeling,’ which serves him as a foundational guide on various types of modelling, including structural, process, and user interface modelling, as well as specialised areas like spatial modelling. Concepts such as conceptual geometric modelling, migration, and Data Integration through ontologies are also addressed in the Handbook of Conceptual Modeling. The material emphasises the importance of conceptual models in conveying our understanding of systems. Howard cites the examples presented in a previous session of electronic circuit boards as forms of conceptual models that facilitate communication about how a system operates.

Figure 1 Example of Canonical Hypergraph

Figure 2 Cover of ‘Handbook of Conceptual Modeling’

Figure 3 Highlighted section on Special Challenge Area

Understanding Data Management and Modelling: Concepts and Literacy

Establishing an implementation framework highlights the critical role of data management in adapting to evolving business needs. Howard recommends paying attention to the interconnectedness of various types of modelling, including data, business processes, user interfaces, and systems. The Zachman framework is referenced because it outlines the vertical decomposition of models from subject area to physical implementation.

Howard shares that most models remain conceptual until they reach coding. The relationship between conceptual models and data governance is highlighted, including elements like Business Glossaries and Data Models. Additionally, Howard emphasises the importance of data literacy by explaining that enabling data citizens to read and understand models involves teaching them the language of these models and the principles of quality assessment, which ultimately facilitates better model creation.

Figure 4 Data Manager Journey Map

Figure 5 Data Manager Journey Map: Focus: “Pot of Gold”

Figure 6 Data Citizen Journey Map

Adoption and Challenges of Data Modelling in Decision-Making

To achieve widespread adoption of modelling for decision-making, it is essential to focus on core concepts while maintaining a balance between simplicity and complexity in developing conceptual models. On this, Howard notes, these models should not attempt to capture every detail, as that can prolong the building process. Instead, the emphasis should be on creating accurate representations that reflect the real world. Howard then speaks on the challenges faced by data professionals in ensuring model accuracy. He encourages sharing experiences related to reviewing and validating Data Models to make informed, data-driven decisions.

Figure 7 Data Citizen Journey Map: Focus: “Pot of Gold”

Figure 8 Data Professional Journey Map

The Data Validation Processes in Economic Analysis

Data validation is a crucial process in data management, especially as it pertains to economic analysis. This involves ensuring the integrity of data as it moves through various stages, capturing any alterations or transformations it undergoes. Key components include verifying that data remains unchanged during transfers, confirming that updates are consistent and logical, and ensuring that metadata definitions align with the actual data. Additionally, thorough statistical validation is necessary to identify and assess anomalies that may arise due to genuine economic events, such as a peculiar spike in trade data due to unusual transactions that require expert analysis to validate. Overall, these validation steps help maintain data quality and reliability for informed decision-making.

Understanding and Implementing Data Concepts

Howard emphasises the importance of quality instance data in databases and datasets, particularly in relation to conceptual models that illustrate relationships among core economic features, such as GDP. A high-level conceptual model might depict these relationships, akin to how consumer price indices utilize weighted averages based on official data series.

Conceptual models are not limited to visual representations; they also encompass verbal descriptions that clarify how various elements interrelate. Such models serve to simplify and communicate complex concepts effectively, as highlighted in examples from fields like biology and environmental science. Ultimately, the accuracy of these models is crucial for data governance, ensuring they accurately reflect real-world dynamics.

Data Modelling: Compatibility, Application, and Branding

Howard shares that this instalment focuses on the compatibility of our Data Model with Master Data and data analytics. He shares that to explore this he must discuss how to create and extract a wide data set for goal-seeking and understanding correlations versus causation among different features. Additionally, Howard states that he will speak about continuous improvement, conceptual modelling best practices, and the effective design of models that accurately reflect business needs.

Additionally, this webinar seeks to address the importance of presenting this conceptual modelling to executives, emphasising the need for organisational buy-in to overcome scepticism from programming teams regarding the value of Data Modelling. Ultimately, Howard seeks to demonstrate how robust Data Modelling can enhance insights, improve decision-making, and foster alignment across the organisation.

Figure 9 Data Executive Journey Map

Understanding the Role and Characteristics of a Theoretical Model

The conceptual model development process involves a Data Modeller who collaborates with stakeholders, including subject matter experts, to create a representation of an organisation, which can be a real-world environment or a set of business processes. The Data Modeller gathers requirements to ensure that the model reflects the core features of the origin and communicates this effectively to various consumers, such as programmers, who need to implement the model based on extracted business requirements. While current modelling efforts, like ORM and ERD, primarily focus on structural aspects, it’s also essential to consider the system’s behaviour through state diagrams that illustrate transitions and actions in response to events, allowing for a comprehensive understanding of how the model reacts under different circumstances.

Figure 10 Essential Concepts Overview

Figure 11 Conceptual Model Characteristics

Challenges and Potential of Concept-Based Programming

The manifesto for conceptual model programming advocates that all programming activities should be conducted at the abstract level of conceptual modelling. It suggests a streamlined process where conceptual models can be directly converted to code and run as applications. However, many have struggled with this approach due to the “semantic gap” between the model, programming language, and the actual application, making accurate conversions challenging.

The emergence of Object Constraint Language highlights the importance of defining constraints and relationships within models, which are essential for proper functionality. Despite efforts like case engineering, which aim to automate software generation from conceptual models, many initiatives have faced failures, underscoring the need for better integration of architecture and Data Modelling rather than solely focusing on immediate coding. Howard mentions Maria Keats’s insights on designing ontology or language specific to conceptual models, which offer potential avenues for enhancing application generation.

Figure 12 Conceptual-Model Programming

Figure 13 Manifesto

Different Modelling Languages and Their Applications in Agile Software Development

Modelling languages highlight a spectrum ranging from code-only approaches to model-only methods. Agile practices often emphasise direct coding, where models may be derived from code later, while round-trip engineering facilitates synchronising models and code by allowing for modifications in both directions. Howard shares on Marco Woben’s demonstration emphasised the importance of maintaining model integrity alongside database changes to prevent model and code drift, which can lead to discrepancies. Model programming advocates for models as primary code generators, whereas model-only approaches focus on understanding system dynamics without producing executable code. Overall, effective modelling is crucial for maintaining alignment between system design and implementation.

Figure 14 The Modelling Spectrum

Spectrum of Data Modelling and Ontology Languages

Howard moves on to discuss various languages used to describe foundational concepts, focusing on taxonomies, declarative languages, and representational languages. Taxonomies organise concepts hierarchically, exemplified by the animal kingdom, allowing for broader and narrower definitions. Declarative languages, as described in Maria Keats’ discussion, include entity-relationship diagrams, RDF notation, dimensional modelling, and more, each serving to model entities and relationships.

Representational languages encompass ontologies and graphical visual languages, such as UML, along with textual representations like the Z notation, which the author studied during their master’s degree in electrical engineering. The Z notation specifically allowed for the description of behavioural systems through mathematical terms, emphasising its utility in modelling state machines.

Figure 15 Conceptual Modelling Language Hierarchy

Types of Programming Languages and Models

Examples of distinctions between different programming languages can be noticed specifically in declarative languages like SQL and imperative languages such as C++. These languages emphasise the importance of UML (Unified Modelling Language) in structuring and managing data through its structural and behavioural diagrams. Structural diagrams, including class, composite structure, and package diagrams, focus on domain organisation, while behavioural diagrams like activity, interaction, and use case diagrams illustrate system operations.

Howard moves on to critique ER diagrams for their lack of behavioural representation, highlighting UML’s advantage in aligning Data Models with application development. Additionally, he clarifies misconceptions about NoSQL, noting that it refers to storage techniques rather than programming languages, contrasting it with various modelling schemes like relational and object-based methods.

Figure 16 Unified Modelling Language (UML)

Figure 17 DMBoK: Scheme (Database Storage) & Notation (Language)

Database Schema in Data Modelling

When selecting a database schema, it’s crucial to choose a structure that accurately represents the data, particularly when working with ontologies, which are best modelled using a graph structure rather than a relational database. While it’s possible to simplify an ontology into a relational format, doing so can lead to a loss of important semantic meanings. Additionally, for NoSQL databases, it’s important to note that they are typically a physical implementation rather than a model designed to handle unpredictable data inputs effectively. Understanding these factors is essential for making informed decisions about database design.

Figure 18 Database Scheme Deciding Factors

Different Types of Data Models and Database Implementations

When selecting a database model, it’s essential to understand that all models, including physical Data Models, are conceptual representations rather than actual instances. For instance, choosing a JSON document does not imply the existence of a JSON conceptual Data Model. Relational models are advantageous as they depict core business rules and concepts, serving as higher-level models applicable across various database storage types, including relational, multidimensional, object-oriented, document, and graph systems. A key example is the entity-relationship diagram (ERD) developed by Peter Chen, which illustrates entities such as Employee, Department, and Supplier, alongside their relationships and attributes. This logical model extends the ERD to encompass attributes and their types, enhancing our understanding of the data structure.

Figure 19 Data Model: Levels of Detail (Abstraction)

Figure 20 Entity-Relationship Diagram (ERD) Pt.1

Understanding the Complexities and Relationships in UML

Unified Modelling Language (UML) extends beyond traditional Entity-Relationship (ER) diagrams by incorporating a wider variety of relationships, including aggregation, composition, and association. This complexity allows for a more nuanced representation of entities, such as classes or tables with attributes and their interrelations. Notably, UML also addresses generalisation, offering concepts like disjoint and overlapping subclasses.

Disjoint relationships can be complete or incomplete, indicating whether all instances can be categorised without overlap—such as distinct categories for male and female employees—or suggesting the existence of additional subtypes. Overlapping relationships allow for instances that can belong to multiple categories, such as an individual being both French and Italian. Overall, UML provides a richer framework for modelling relationships and subtypes within a system.

Figure 21 Entity-Relationship Diagram (ERD) Pt.2

Extension of the Entity-Relationship Diagram (ERD) Language

The development of extended Entity-Relationship Diagrams (ERDs) addresses limitations in traditional ERD language, which was deemed too restrictive for complex Data Modelling. This new conceptual framework incorporates different types of inheritance, complex attributes, and aggregation, elevating the level of detail in data representation. Additionally, it introduces rules for calculations, such as currency and exchange rates, while also covering participation constraints, both mandatory and optional, which relate to cardinality and optionality. This extension allows for a more comprehensive understanding of data relationships and structures within UML while maintaining the familiar ERD diagram format.

Figure 22 Extended ERD

The Nuances of Dimensional Modelling

Dimensional modelling is crucial for effectively organising data, and understanding its conceptual framework is key. The Peacock diagram or axis notation serves as a visual representation of the grain of a dimensional model, illustrating the hierarchical relationships between different dimensions, such as time (from year to month) and product (from category to line to product), that connect to transactions like orders. Properly defining the grain is essential, as an incorrect grain can lead to user frustration—especially when they seek to drill down to more specific data, such as daily figures, but are only provided with aggregated monthly data. This highlights the significance of accurately modelling data to meet user needs.

Figure 23 Dimensional Models

The Implementation of focal point Modelling in Data Warehousing

Focal point modelling, specifically through Data Vault modelling, emphasises a structure consisting of hubs, links, and satellites. A hub represents a core entity, characterised by a single attribute—typically the customer key—while additional attributes are stored in satellites. This design simplifies the process of adding new customers to the Data Vault, as only the customer code is necessary for registration.

Relationships between entities, such as customers and products, are managed via links that detail these connections. Data Vaults are categorised into a raw Data Vault, which integrates incoming data, and a business Data Vault, which organises it conceptually. Ultimately, data vault modelling supports traditional dimensional modelling, acting as a modern replacement for enterprise data warehouses and the third-normal from relational databases.

Figure 24 Data Vault

Anchor Modelling and Database Normalization

Anchor Modelling represents a database design approach that achieves sixth normal form (6NF) by treating every attribute as a separate table, allowing for great flexibility and extensibility in data management. This method incorporates historized attributes, enabling the tracking of changes over time, such as name variations for specific entities. Unlike traditional relational databases that require downtime for modifications, anchor modelling is additive, allowing for seamless updates without disrupting the system. Recent advancements in database vendors like SQL Server have improved access plans, enabling efficient data retrieval without the need for extensive denormalisation, thus addressing performance issues historically associated with fully normalised databases.

Figure 25 Anchor Modelling (6NF)

Understanding focal point Modelling and Unified Decomposition

Focal point modelling, also known as unified decomposition, centres on identifying the core concepts of an entity, such as a customer, which is highlighted as the main focus. This approach involves creating an ensemble, which consists of additional entities that support and enrich the core concept, akin to the satellites in Data Vault modelling. For instance, while a customer is a primary entity, details like an address are considered secondary and not as essential. By understanding the hierarchy and importance of these entities, one can effectively decompose them into their components and reconstruct them into dimensions for dimensional modelling, thereby creating a cohesive and organised data structure.

Figure 26 Focal Point Modelling

Advantages and Differences of Fact-Oriented Modelling in Business Data Management

Fact-oriented modelling, including Object-Role Modelling (ORM), emphasises understanding business facts by engaging directly with stakeholders to identify key elements such as providers, travel details, and transportation codes. This approach highlights entities and encompasses all relevant data within the model, making it easier for business professionals to comprehend and validate the information presented.

By focusing on practical communication and avoiding excessive abstraction, fact-oriented modelling fosters a clear “universe of discourse” that aligns with the business language. Comparisons between ORM, Entity-Relationship Modelling (ERM), and Ultimate Modelling Language (UML) can further enhance understanding, although time constraints may limit in-depth discussions.

Figure 27 Fact-Oriented Modelling (FOM)

Figure 28 Example of a Fact-Oriented Model

Understanding Database Models and Their Applications

Howard outlines the differences between various Data Modelling approaches, particularly focusing on ORM2 and FCO-IM. He discusses NoSQL databases, highlighting the advantages of using JSON documents over traditional relational databases, particularly the flexibility to add attributes without needing to modify the table structure or consult a DBA. However, this flexibility can lead to a lack of transparency regarding the database content, often requiring tools like Idera for reverse engineering of stored data. Additionally, knowledge graph databases, which incorporate metadata for relationships among entities, are distinguished from property graph databases, providing context about connections, such as between people, artworks, and locations. Overall, the discussion emphasises the evolving landscape of data storage and retrieval methods.

Figure 29 Method Comparison

Figure 30 ORM2 or FCO-IM

Figure 31 JSON

Figure 32 Document Database

Understanding Semantic Modelling and Data Storage

Semantic modelling involves creating a conceptual framework that illustrates the relationships between various concepts without relying on physical storage. For instance, in the context of food, items can be categorised by their components, such as sugar, amino acids, or other ingredients, which collectively define a food group.

To determine the appropriate model for a specific scenario, it is crucial to understand the type of data the business requires. For example, if the data is unstructured and consists of documents, a graph database should be utilised instead of a relational database to effectively store and interpret the information by breaking it down into individual elements categorised by subject, predicate, and object.

Figure 33 Knowledge Graph

Data Storage and Decision Making in Business

To effectively support business decision-making, it is crucial to first identify the types of data required and determine the most suitable storage mechanism. For a graph database, consider employing a knowledge database or ontology to describe document content. Alternatively, for image or video data, a key-value pair storage approach may be ideal, utilising BLObs (Binary Large Objects) to store and retrieve media without needing detailed metadata. It’s essential to focus on the specific business requirements regarding data types and storage preferences while also integrating inference engines to enhance decision-making capabilities. A knowledge graph can be particularly beneficial in this context, facilitating insightful analysis based on the data.

Concepts and Challenges in System Modelling and Data Virtualization

Howard shifts the focus to the challenges of defining accuracy within a conceptual model, particularly in relation to data virtualisation techniques. He highlights the potential of achieving ultimate accuracy through a modelling manifesto, where modifications to the model can generate a virtual Data Model that interfaces with various underlying databases for real-time querying.

While data virtualisation simplifies integration by avoiding the need to generate application code, it limits the ability to create a fully functional system for testing purposes. The aim is to develop a system that allows for generated code and user interface testing to evaluate the effectiveness of the conceptual model, which, unfortunately, remains unfulfilled at this time.

Figure 34 Semantic Modelling

Importance and Implementation of Data Modelling Evaluation Boards

To measure data accuracy, it’s essential to create a set of scorecards that evaluate various aspects such as business definitions, Data Modelling, data flow, business processes, user interfaces, and model-driven development. For instance, the data flow scorecard assesses whether the implemented data lineage matches the design specified by data architects through a process called stitching.

A notable example of a Data Model Scorecard was developed by Steve Hoberman, which evaluates factors like correctness and completeness using a systematic approach. This method emphasises the importance of thorough preparation before peer reviews and encourages individuals to address identified issues before re-evaluating. Additionally, utilising tools like Power BI can help track improvements in Data Modelling skills over time and identify recurring challenges.

Figure 35 Model Accuracy

Figure 36 Data Model Scorecard

Figure 37 Model Compatibility

Figure 38 “Correctness Rules”

Figure 39 “Structure Rules”

The Integration of Data Models in Various Domains

Data Models play a crucial role in various domains, such as Reference and Master Data, party information, product details, and finance. Effective Data Modelling requires the creation of taxonomies, hierarchies, and integration approaches, including canonical models and data virtualisation techniques. Essential components include database schemas, data dictionaries, data warehousing, and self-service data science practices, alongside metadata management.

To ensure coherent data management, it’s vital to develop specific model types for reference data, integration, and storage, enabling a comprehensive view that facilitates impact analysis. The use of a unified model allows for changes—such as updates to business definitions—to be systematically addressed, ensuring all interrelated models are aligned and modified as necessary.

Importance and Implementation of Data Architecture

The conceptual modelling playbook should encompass a comprehensive policy for modelling, an implementation framework, and a procedure outlining the roles of input suppliers and output consumers for each deliverable. A scorecard, previously shared, is also essential to measure progress. It’s crucial to conduct this planning at the data architecture level, as it guides the design of both Data Models and meta models. Key considerations include integrating various Data Models, understanding data types, assessing risks, and analysing the current data landscape, followed by the development of platforms, data solutions, integration strategies, and data products. This approach ensures a cohesive framework for data architecture and modelling.

Figure 40 Continuous Improvement: Conceptual Modelling Playbook

Figure 41 Data Architecture Implementation Framework

Figure 42 Implementation Framework SIPOC

Collaborative Practices for Concept Modelling

When creating a conceptual model, it’s essential to understand its objectives, engage the right stakeholders, and use appropriate notation. Determine whether you need to describe processes, data, user interfaces, or application logic and focus on both written and visual representation. Concept maps are effective for compiling a business glossary and defining terms, while conceptual models serve for visual presentation. Ensure that metadata is accessible, allowing users to quickly comprehend the Data Model, its lineage, and potential impacts. Additionally, validating the model through a fact-driven approach can help stakeholders confirm its clarity and relevance, making it easier for business SMEs to affirm that the model aligns with their understanding.

Figure 43 Conceptual Modelling Best Practices

Figure 44 Advanced Techniques Decision Science

Art and Science of Decision Making

Decision science encompasses advanced techniques for effective decision-making in business by focusing on key aspects such as representing the decision process, managing uncertainty, and incorporating behavioural insights. Essential elements include decision modelling, which utilizes decision trees to graphically depict specific decision-making processes, and influence diagrams to illustrate relationships between decisions and uncertainties.

Understanding data quality is crucial for distinguishing causation from correlation, while multi-criteria decision analysis helps address conflicting priorities. By identifying the various components of a decision, organisations can pinpoint weaknesses and enhance their overall decision-making strategies.

Figure 45 Advanced Techniques Decision Science

Advanced Analytics and Dimensional Modelling in Business Intelligence

The discussion focused on advanced analytics, particularly emphasising dimensional modelling and its role in creating data marts for business intelligence (BI). Dimensional modelling facilitates quick aggregation of data, akin to pivot tables in Excel, allowing users to analyse sales metrics over various time dimensions seamlessly. It serves as a performance-enhancing structure by providing pre-aggregated figures at different levels, such as year or month, which expedites query responses. In contrast, more detailed ad hoc queries are typically executed against the business data vault, where the relational structure is easier to navigate. The conversation also touched upon the necessity of selecting appropriate models based on the type of analytics being performed, particularly in the context of data science and AI.

Figure 46 Conceptual Modelling for the Data Executive

The Use of Graph Databases and Vector Relationships in Data Modelling

A graph database can be utilised for logical inference or deduction of information, with examples like LOMs employing vectors to represent relationships between words. This involves using both vector and graph databases to connect words based on their statistical relationships, enabling capabilities such as predictive text generation. For instance, when typing in tools like Copilot, the system constructs a statistical model that interprets the words inputted, thus understanding the context and predicting subsequent words by examining the proximity of related terms. This method highlights how words are linked through their relationships, facilitating intelligent suggestions based on data ingested from various sources. Additionally, the conversation touched on the importance of Data Modelling and its compatibility across different knowledge areas, emphasising the need for effective data management strategies.

Understanding Wide Data Sets and Feature Engineering in Data Science

Wide data sets are pivotal in data science, particularly for feature engineering and goal-seeking. They store extensive data—often in the range of thousands of columns—allowing analysts to examine the relationships between various features to achieve specific outcomes. For instance, by combining attributes from different dimensions and fact tables within data marts, one can create a comprehensive table that encapsulates diverse variables such as age, gender, year, product, and region. This integrated approach enables a deeper understanding of how these elements relate to one another, facilitating a more effective analysis of behavioural patterns.

Data Modelling and Integration for Different Data Analysis Needs

Howard highlights the importance of Data Modelling and integration across various domains to create structured environments suitable for different types of analysis, such as inference and deduction through knowledge graphs that require ontologies and taxonomies. For numerical analysis, unstructured data needs to be transformed into tabular formats, akin to Excel sheets, to enable calculations like sums and averages.

NoSQL databases excel in storage, and I/O operations require re-engineering them into rectangular data shapes for analytical purposes. Essential elements like metadata define the location, meaning, and format of data, enriching Data Models with a business glossary that standardises terminology for better understanding. Ultimately, to facilitate calculations, independent documents must be aggregated into a single database, allowing for effective data virtualisation and querying.

Evolution and Efficiency of Document Databases in Digital Services

Howard wraps up the webinar with an anecdote of Amazon which faced challenges with its relational database when handling product queries due to excessive Input/Output (IO) operations required for retrieving information across multiple tables. To improve efficiency, they adopted a document-based approach using the JSON format, allowing them to return a single document with all relevant information for a query. Initially successful, this method encountered scalability issues as document sizes grew, reaching up to 3 gigabytes for a single screen. In response, Amazon implemented document linking to create partial loads, which improved performance while still enabling better data management than traditional relational databases. Similarly, platforms like Craigslist utilise non-structured document or key-value databases to store various types of content, demonstrating the limitations of relational databases for specific applications.