Data Abstraction for Data Professionals

Executive Summary

This webinar covers the role abstraction plays in both vertical and horizontal data modelling, and why both matter so much for effective data management. It works through why precise definitions and clear presentation matter for successful communication in business analysis and enterprise design. Howard Diesel outlines the process and levels of data modelling, and why operational and specification models matter within enterprise architecture. He also covers spatial presentation, concept mapping, and building subject areas in the data landscape, all in service of better understanding and knowledge management within international organisations.

Webinar Details

Title: Data Abstraction for Data Professionals
Date: 21 July 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

Abstraction for Vertical and Horizontal Data Modelling

Abstraction is key to understanding complex ideas. It’s a general concept that shows up in a variety of contexts, all in service of simplifying information and making it easier to grasp. There are different types worth knowing: vertical abstraction, for instance, drills down into detail from higher-level entities and relationships toward more detailed information. Reverse engineering is a good example, starting from the database and building up logical and conceptual models. There’s also the middle-out approach, which works with the conceptual model and subject areas, moving from high-level to detailed information. Abstraction for presentation, meanwhile, is about simplifying concepts, lines, and information for effective communication with the business, and there are abstractions for specification, common attributes, and subtyping too. In the end, abstraction’s real job is helping people understand complex ideas by focusing on what’s essential and stripping out what isn’t.

Figure 1 Dictionary Definition: Presentation

Clarification on Horizontal and Vertical Abstraction for Presentation

Different levels of abstraction run from whole specification through executive management and architect-engineer levels down to data models themselves, and the discussion centred on how those various levels of a data model actually play into communication.

JG raised a question about how a data estate connects to a subject area model, and at what point horizontal abstraction actually transitions into vertical.

Abstraction in Data Management

An organisation’s data estate is really a collection of the business entities that matter most. Within it, data areas help build out a data strategy and identify touch points and data ownership, and metadata gets organised around subject areas, which also get used in master data domains, assigning ownership and keeping security intact. Abstraction here isn’t about shared characteristics or connections, it’s about presenting information to the right audience, and there are three types in play: specification, presentation, and commonality.

The Importance of Precise Definition and Simple Presentation in Writing

Discussing a conceptual model means staying clear in both definitions and presentation, so readers can actually understand and agree with what’s being shown. Understanding what a concept actually means matters a great deal here.

Some might treat concepts as critical business entities, but it’s worth being clear that they’re not the same as critical business or critical data elements. A concept is a refined notion, shaped through extensive analysis and discussion, while an idea is more of a rough mental construct.

Building accurate business definitions matters before conceptual models even get started. Ideas tend to be individual efforts, while concepts need agreement from a group, business departments especially.

Figure 2 Dictionary Definition: Concept / Conceptual

Abstraction in Data Modelling

Leaning solely on personal ideas or external models when building a data model can get in the way of reaching group agreement, so it’s worth avoiding. In business communication, everyone needs to recognise and agree on the basic concepts involved.

Abstracting too high when building a master data model can cause real problems, so it’s worth watching that. Abstract patterns are still necessary, though, for handling multiple entities and information effectively.

Conceptual models in data architecture describe and evaluate objects or actions tied to business transactions or events, using abstract concepts to explain theories and relationships along the way.

Conceptual models can be broken down into subject areas, with logical and physical models for each application following from there. There are two approaches to abstraction in data modelling worth knowing: horizontal abstraction, defining each subject area one by one, and vertical abstraction, drilling down from conceptual to logical and physical models.

The horizontal abstraction approach, sometimes called middle-out abstraction, defines and evaluates subject areas horizontally to surface dependencies between them.

Figure 3 Enterprise Data Model

Vertical and Horizontal Abstraction in Data Modelling

Analysing a database’s data model comes down to two types of abstraction: vertical and horizontal. Vertical abstraction means delving into the details, reverse engineering included. Horizontal abstraction, on the other hand, groups data into distinct subject areas, product design, commercial offerings, sales, and those subjects can then be examined more closely for detailed modelling. Building a commercial offering, for instance, needs both product design and sales concepts defined, with the conceptual model taking shape by integrating concepts from each subject area. Horizontal partitioning can help too, letting you focus on specific parts of the diagram for deeper analysis.

The Importance of Vertical and Horizontal Data Modelling in Enterprise Design

Data modelling really splits into two main approaches: vertical and horizontal. Vertical data modelling focuses on a specific subject area or data model, while horizontal data modelling takes in the bigger picture of an Enterprise data warehouse.

Global design is a key part of data modelling, building horizontal abstractions to understand every element of the data estate. Local implementation, meanwhile, happens when a specific area, product design, say, gets picked out for vertical implementation.

During vertical implementation, conceptual, logical, and physical models get built one subject area at a time, working from the broadest conceptual level down to the most specific physical one. The goal, in the end, is getting the data model into the transactions themselves and building models that are genuinely solid.

Worth noting: conceptual, logical, and physical models typically operate at enterprise level, but there are lower-level models too, specific to individual applications or projects.

Figure 4 The Importance of Vertical and Horizontal Data Modelling in Enterprise Design

The Role and Process of Data Modelling

Data management really relies on two important roles: the data architect and the data modeller. The data architect builds the Enterprise model, while the data modeller focuses on a model for a specific application or data product.

The data modeller can build a conceptual model too, though it’s not always necessary if a solid Enterprise data model already exists. Either way, the data modeller is responsible for the logical and physical models for that application or data product.

Steve Hoberman’s scorecard is worth keeping in mind at every level of the models, for accuracy and effectiveness, and communication matters just as much, particularly when scope and requirements are being discussed across different levels.

Building a conceptual model means highlighting concepts straight out of the business requirements in a Word document, and it’s worth making sure any concept added to that model actually falls within the scope of those requirements.

Importance of Clear Communication for Business Analysis

Clear communication matters a great deal in business, and that means giving a precise definition and layout of business models, then checking that businesspeople actually understand and feel comfortable with the proposed model. Horizontal and vertical drill-down methods help with deeper analysis, and visually representing client requirements along the way is worth doing too, with examples used to illustrate the steps of the analysis process. Getting agreement at a high level before diving into detailed analysis matters more than anything else here.

Figure 5 Abstraction: Horizontal / Vertical

Levels of Detail in Architectural Modelling

The architect develops a conceptual model to show the design’s overall look and feel, which then gets transformed into a logical model with details like room layouts and door placements. That logical model becomes a physical model that guides the electrician on the lighting, and that progression is what ensures the design gets thoroughly understood and executed properly.

If a technical implementation misses certain concepts, it may need reverse engineering to bring it back in line with the initial requirements. Depending on what’s required, different databases can support different physical implementations too, JSON, relational, or graph among them.

Abstraction here really means presenting data at different levels of resolution, rather than aiming for generality or commonality.

Figure 6 Levels of Detail in Architectural Modelling

The Importance and Use of Abstraction in Data Modelling

Understanding the context abstraction is being used in matters a lot for avoiding confusion, and different models get used in data modelling to present and communicate with the audience effectively. On the data modelling exam, questions on super type subtype, generic generalisation, and horizontal and vertical abstraction carried real weight. Abstraction shows up in plenty of areas beyond modelling too. For specification specifically, it means creating a data sheet with specific information for a particular purpose, while vertical abstraction illustrates high-level subject areas through conceptual, logical, and physical models. Linking the conceptual model back to the business process model, logistics, and workflow matters throughout data modelling.

The Importance of Operational and Specification Models in Enterprise Architecture

Understanding the annual operations model is crucial for your timing model and business plan to succeed. TOGAF creates a range of processes and deliverables for enterprise architecture. The Zachman framework highlights the core elements of enterprise architecture through what’s known as the Periodic Table of models, covering different areas and processes. Deliverables need to align with the vertical organisation’s needs and specification model, and system logic brings inventory, process, and organisational representation together. Different levels of detail get presented to keep things understandable, with irrelevant information stripped out, and the specification element is what confirms the client’s actual needs. Stripping data vertically or horizontally either way is what provides the detail needed for implementation.

Figure 7 Data Architecture Framework Abstractions

Importance of Data Architecture in Data Management

Effective data management within an organisation depends on a well-designed data architecture, which means representing data at different levels of abstraction to handle large volumes of it.

The subject area model is one good way of allocating responsibility, data ownership and stewardship included, giving a comprehensive view of core business concepts with no gaps or overlaps.

Balancing a global perspective with local implementation matters a great deal when building an Enterprise data warehouse, and the terms “subject area,” “data estate,” or “landscape” can be used pretty much interchangeably to help with comprehension.

Executives can sometimes struggle with the subject area model as a term, though, which is why alternative terminology gets used. A high-level data diagram helps visualise data hierarchy too, though confusion can still creep in between conceptual and logical terms.

Data Modelling Levels

This presentation walks through a high-level view of a house using various data models. First comes the subject area model or data estate, giving an overview of the entire house, with a simplified high-level data model and basic definitions showing the house, porch, and bedroom. As the discussion goes deeper, a more detailed conceptual data model comes in, bringing specific business rules and definitions with it.

A logical data model shows up as a floor plan, with attributes like doors and wash basins included, and finally a physical data model, represented as a wiring diagram, shows the information reduced to the appropriate level.

Figure 8 Data Modelling Levels

Understanding the Process of Data Modelling

Moving toward higher levels of abstraction means discussing presentation and imported elements, while moving down the levels means gathering more detail while still retaining vertical abstraction.

A concept, essentially, is an idea or notion that’s been developed collectively, and conceptualisation itself is about defining relationships and abstraction.

Logical data models are built to follow logic and sound reasoning, and every model should have a clear rationale behind its design.

Specific details may be concealed during the presentation phase to communicate with users efficiently.

The data modelling process starts with understanding the business scope and gathering user narratives, which then get converted into sentences or facts, with nouns and verbs doing a lot of the work.

Business fact modelling is really about remodelling the facts about the business, while ER modelling focuses on primary keys, column names, attributes, normalisation, constraints, and foreign keys.

Relational modelling is just one aspect of the overall data modelling process.

Figure 9 Data Model Levels Created by Abstraction

The Process of Data Modelling and Abstraction in Database Design

The process starts with gathering information from the business, facts, nouns, constraints, types, then creating a relational model through denormalising, resolving keys, and defining attributes. From there, it’s on to a business data model and multiple conceptual views for the business to approve, alongside SQL code and data definition language (DDL) provided to developers and DBAs for internal use. Logical, physical, and conceptual modelling all get used together to keep communication clear, and appropriate views get selected for the business based on what they specifically need.

The ultimate goal is representing the business requirements effectively and adding real value to their operations. The levels of modelling in database design aren’t so different from spatial representation in traditional architecture.

Figure 10 Presentation / Levels / Stages

Importance of Spatial Presentation in Data Modelling

A well-defined spatial layout matters when building data models, not unlike how rooms get laid out in a house, but establishing clear relationships between the data models matters even more. Lower-level diagram models need to stay consistent and free of conflicts, and since data moves both horizontally and vertically in data modelling, traceability from the data dictionary through to the physical and logical models is essential. Clear diagramming when structuring logical models is what keeps things readable, and following specific rules, starting with essential points on the left-hand side and reading clockwise, for instance, improves both layout and understanding. Getting that diagramming clarity right is really down to Data Architects defining the practice of data modelling properly.

Defining Business Practices and Knowledge Management in an International Organization

Every business needs its own approach to operations, and Steve Hoberman has a method for outlining and communicating business practices that helps here. The emphasis is on clearly defining an organisation’s models so they get interpreted accurately, and camera settings work well as an analogy for establishing standards and focus, in the same way an artist directs a viewer’s attention to the subject that matters, extending naturally to layout too. Categorising data states helps systematise and present data efficiently, and visual tools help simplify complicated concepts, not unlike constructing a house.

Concept Mapping and Graph Databases for Data Modelling

Using Power BI in data flow development makes it possible to visually represent how data sources affect reports. Tools like concept mapping and graph databases help simplify comprehending and mapping data relationships, with concept mapping working dynamically with a concept map to visualise and understand business relationships, and graph databases highlighting related nodes to a particular concept, useful for applying normalisation too. Concept mapping modelling is a genuinely effective way to build data models and interact with data visually, and it’s worth constantly weighing both top-down and bottom-up approaches to data modelling, since that’s what surfaces new insight and a better understanding of how elements relate.

Creating Subject Areas in the Data Landscape

Mark Atkins has built a tool that uses a graph database to link business definitions with terms, which is how subject areas get created. Unlike Informatica Collibra and similar tools, Atkins’ tool offers visualisation capabilities, which is a real advantage. Standardised data modelling tools take a very different approach when it comes to generating new ideas for modelling data and concept maps. Marco Woburn’s fact modelling technique, meanwhile, centres on linking facts with entities and relationships to represent the data landscape visually. Grafana is another tool that comes up often in the context of visualising subject areas, and business capability models can also help identify subject areas within the data landscape.