Data Abstraction for Data Executives

Executive Summary

This webinar covers abstraction and generalisation in data modelling, two ideas that matter a great deal for building effective models. Abstraction simplifies complex data, while generalisation groups things into broader categories, and understanding both is central to modelling well. It also covers abstraction safety in writing models, being aware of the pitfalls and how to guard against them, along with normalisation, flexibility, and complexity. Cataloguing and meta-models get their own coverage for organising and managing data, as does testing and questioning models for accuracy. A large part of the discussion focuses on the party model specifically, a model built for business transactions that can be genuinely valuable to build well, along with industry data models more broadly. The session closes by working through the different types of data models and the various abstractions that go into modelling successfully.

Webinar Details

Title: Data Abstraction for Data Executives
Date: 28 July 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

Abstraction and Generalization in Data Modelling

Abstraction is about building data models that are both more general and more specific, though that approach carries real risk, weaker enforcement of business rules and time wasted on elements that don’t actually matter. How much abstraction a model needs varies by type; the logical model tends to offer the most room for generalisation, while models like the dimensional model call for more caution, since concrete representations are usually preferred there. Regardless, abstraction plays a role at every level of data modelling, conceptual, logical, and physical, and it’s what makes it possible to extract and work with information effectively.

Figure 1 Dictionary Definition of Abstraction

Abstraction in Data Modelling

Abstraction in data modelling is really about simplifying complex information, focusing on what matters and stripping out the variables that don’t. It applies across several areas: presentation, specification, interpretation, and data modelling itself.

Data modelling abstraction means moving from concrete to abstract terminology, going from an individual to a party, say, or a product to a thing. There are a few types worth distinguishing: visualisation abstraction, which strips out variables to get aggregated data and see how terms reflect against each other, and data model abstraction itself.

Communicating clearly with the business matters a lot here, making sure they actually understand the concepts being presented, since too much abstraction can leave business stakeholders lost.

Zooming in on details gives you a concrete implementation, while zooming out produces a more abstract version, and knowing when to do which is an important technique in its own right.

Data modelling covers a range of model types, application data models, dimensional data models, metadata models, master data models, and each calls for a different level of abstraction. Dimensional models and reporting systems tend to reject abstraction outright, while metadata models are built around entities, attributes, and relationships. Getting the level of abstraction right, in the end, is what lets a model actually represent and make sense of complex information.

Understanding Data Modelling Abstraction

Building an ontology depends on abstraction to establish the concept of a “thing” in the first place. Different models call for different levels of it, and the choices made during modelling shape the results you get. In practice, abstraction transforms attributes, relationships, and entities into more generalised structures: simpler models might stick to entities like “supplier” or “customer,” while more complex ones, Data Vault among them, lean toward physical data models for data warehousing. Worth noting: abstraction shouldn’t be used in reporting with Data Vault modelling. Master data models tend to sit a bit further toward the abstract end, and meta-models further still.

The Importance of Abstraction and Abstraction Safety Guide in Writing

The Abstraction Safety Guide, Steve Hoberman’s concept, was built to address exactly the problems that come from over-abstracting, aiming for simplicity, flexibility, and quality in the writing itself. Normalisation is the first step toward flexibility, and the abstraction process itself works through attributes, entities, and relationships. Documentation matters a great deal throughout, and the normalisation view, the abstraction, and the documentation all need to come together. A key part of abstraction is making sure there’s real commonality across attributes and elements, and Howard’s advice is not to keep abstracting once you’re no longer finding matches in the model.

Abstraction and Normalization in Data Model Development

Deciding whether an abstract structure is actually needed comes down to weighing real-life scenarios against the benefits abstraction offers. If it’s genuinely more efficient to invest extra development effort now rather than make changes later, abstraction is probably the right call.

Abstraction pays off particularly well for SAP models and application models that need ongoing updates, since it gives you flexibility for handling unique customer needs. Both abstraction and normalisation are aiming at flexible data models, just from different angles: normalisation reduces redundancy and maximises flexibility for current business requirements, while abstraction is more about accommodating future requirements and new types as they come along.

Subtype and supertype relationships in databases need careful thought, and often involve roll-up and roll-down identity. A table showing redundancy across various phone number types is a good example of something that could be addressed through either normalisation or abstraction.

The Importance and Challenges of Abstraction in Data Modelling

Meeting future requirements takes a flexible model, and one way to build that flexibility in is through an abstract data model.

A customer phone abstraction with phone types illustrates this well. Understanding the difference between normalisation and abstraction matters here, and common attributes help surface similarities; adding priority numbers to phone contacts is a useful touch too.

Abstraction brings real benefits for application stability and integration. More abstraction in the Enterprise data model, for instance, can cut design time, and using roles along with start and end dates supports consistent identification.

New opportunities open up across different types of customers and products through typed communication and marketing channels, and the Enterprise data model steers clear of lower-level business rules. That said, there are real challenges too: models can end up vague and imprecise, and there’s a real risk of losing business customers along the way.

Quality enforcement and consistency can slip in the database model too, and more abstraction can bring performance issues along with it. On top of that, justifying abstraction’s long-term benefits to a project team isn’t always an easy sell.

The Challenges and Benefits of Abstraction in Data Modelling

Generating accurate reports gets harder when reporting tools are working off generic fact tables in data warehousing. Assigning ownership in data modelling has its own pitfalls too, stewardship gaps tend to open up when people are reluctant to take responsibility.

Tacticians tend to be good at knowing when abstraction is actually needed for action, while strategists will often use it even when no action is required. Data modelling leans on both patterns and abstractions, and generally, the higher the level of abstraction, the simpler it gets, while lower levels bring more complexity.

The assembly pattern is a good example of a mutually exclusive, collectively exhaustive pattern used in data modelling. At the entity level, elementary attributes bring their own distinct kind of complexity.

Flexibility and Complexity in Data Modelling

Modelling a product means working out its various features, which can carry a huge range of potential attributes. Including elementary attributes makes the data model more adaptable, but also more intricate, since it lets you add virtually any attribute to the product entity.

That complexity can be genuinely hard for business users to work with, and different attribute typings can make reporting and aggregating data trickier too. Which is why building in different levels of abstraction matters, it’s what gets you the flexibility you actually need.

Subtyping and supertyping can complicate things further still. Even so, flexibility matters a great deal for transactional and product systems that let customers customise products with their own unique attributes.

Figure 2 Flexibility in Data Modelling

Figure 3 Complexity in Data Modelling

Cataloguing and Meta Models in Writing

Howard worked through why cataloguing matters as a fundamental way of organising and classifying information, providing a baseline standard for how information gets classified. ISO standards commonly get used in cataloguing and can tie into the system being discussed. Building master data domain models for products means weighing various characteristics and building in flexibility, and David Haye’s book on modelling patterns explores different levels of abstraction in cataloguing specifically. A meta-model lets you link data from different tools together for impact analysis and relationship building, and the ultimate meta-model can pull metadata elements in from any model, stringing everything together as one vertical abstraction.

Figure 4 Cataloguing and Meta Models in Writing

The Importance of Testing and Questioning in Model Abstraction and Meta-Models

Testing and validating complex models matters for surfacing ambiguities and inconsistencies, and patterns along with past experience help make sense of and test abstract models. Testing meta-models is genuinely hard, and it’s worth questioning the results of any change made along the way. Informatica does this well, collecting metadata from different tools and using abstraction to get better flexibility out of it. Meta-meta models are increasingly in demand for pushing abstraction even further. Master data models work well as domain-specific abstractions that are genuinely usable, and abstraction brings real benefits within a master data system, with typing abstraction playing a particularly important role in party data models.

Understanding the Party Model

Howard first came across the party model about fifteen years ago, while building an application. One of the business team members struggled to follow the model at the time, so Howard came up with an analogy, “the coat hanger model,” to make it click.

The party model is really a system for holding information about a party, whether that’s an individual, a business, or a bot. Think of each attribute of a party as a hanger, with relevant information attached to it.

Parties get uniquely identified through ID numbers or passports for individuals, and commercial registration numbers for organisations, and depending on how an organisation is structured, a party’s location can vary across different areas and places worldwide.

Figure 5 Understanding the Party Model – The Coat hanger Analogy

The Importance of Building a Party Model for Business Transactions

Whether you’re communicating with individuals, bots, or website locations, it matters that you know clearly who you’re actually dealing with, which is what a centralised system holding all the relevant information, names, bank accounts, addresses, gives you. Parties can hold various roles in a business transaction, and integrating them into transactional systems is a straightforward way to work that out. Building a party model in a master data system consolidates all party-related information, which supports better data privacy management, and it also keeps things consistent and makes individuals or entities easier to identify. The key principle is centralising party elements and linking external subject areas back to parties using a single ID.

Figure 6 The Importance of Building a Party Model for Business Transactions

Notes on Insurance Data Model and Party Model Concepts

Howard has reviewed an insurance data model and found a fair number of flaws and data quality issues in it, suggesting the fix is simplifying things down to a single-party ID instead of juggling multiple identifiers for insured and producer parties. He walked through a party model’s key elements: characteristics like name and location, identification, and other attributes, along with why classifications, associations, and grouping matter for understanding how parties relate to each other. The party model, he stressed, is really what pulls scattered attributes together cohesively, and using nicknames, preferred names, and registration names properly is a genuinely important consideration within it.

Figure 7 Party Model Creep

Figure 8 Core Elements of the Party Model

Various Data Models and Their Components

A few different data models matter for managing data well: the IBM Model, the Oracle Party Data Model, and the Universal Data Model among them. Lane Silverstone has contributed a great deal to the latter, particularly around the patterns and party concepts he introduced.

Flexibility is another key consideration, genuinely vital in today’s fast-moving business environment, and bringing in different elements and roles is how you build more of it into better data management.

Object-oriented implementations of data models are picking up traction too, with the Object Management Group leading that charge. The Financial Industry Business Data Model (FIBO) is worth attention as well, integrating various roles and contract components.

Informatica’s Data Privacy Manager is a genuinely useful tool for safeguarding customer data and staying GDPR-compliant, and Oracle’s data integration capabilities have proven highly effective for bringing data together and managing it well.

Figure 9 Oracle Party Data Model

Figure 10 Universal Data Model – Party Model

Figure 11 Making Master Data Personal

Notes on Data Modelling Abstraction

Turn on the Oracle Master Data Hub or Customer Hub, and the configuration and authoring of customers within the Oracle application itself gets disabled. Instead, customer authoring shifts into the master data system, with the customer list populated from the Customer Hub rather than its own tables, which makes harmonising applications as simple as flipping a switch.

There’s sometimes a case for separating location out into its own model within the Party data model, along with other attributes like communication methods. Abstraction brings flexibility and simplicity, but it’s worth being careful with it, since it can introduce duplication and data quality issues in transaction systems if you’re not.

Figure 12 Oracle Customer Hub

Understanding the Different Abstractions in Data Modelling

Howard worked through the various levels of specification and abstraction in data modelling: presentation, vertical abstraction, data modelling abstraction, and interpretation. What these abstractions are ultimately working toward is converting business requirements into a concrete database, while making the changes between the two clear along the way. Abstraction shows up in different scenarios too, data to physical, motivation, conceptual modelling, and Howard’s closing point was how much it matters to understand and spell out exactly which context the term “abstraction” is being used in.