Data Abstraction for Data Managers

Executive Summary

This webinar makes the case for how central abstraction and specifications are to system modelling and software engineering. It works through what a specification actually is and why it matters, and why writing clear business definitions matters just as much. It also covers why functional and non-functional requirements need to be kept distinct, along with the role visual presentations and data modelling play in conveying the essential aspects of a system.

Howard Diesel breaks down the role of analysis artefacts and data life cycle management alongside agile development practices like test-driven development. He makes the case for clarity, avoiding unnecessary detail, and for defining critical data elements as a way to keep project scope under control from the conceptual stage. Altogether, the webinar builds a solid understanding of best practice for developing robust system logic and process models.

Webinar Details

Title: Data Abstraction for Data Managers
Date: 25 July 2023
Presenter: Howard Diesel
Meetup Group: African Data Management Community Forum
Write-up Author: Howard Diesel

Abstraction and Data Models

Howard Diesel opened the webinar by pointing to the benefits abstraction brings to master data projects, helping create conceptual models and simplify data down to the essentials. It’s worth understanding the problems abstraction can cause too, though, particularly in relational data models.

There are several ways to abstract: removing characteristics, using shapes and patterns, simplifying for presentation, or considering general qualities apart from concrete realities. Whether to use absorption or partitioning in data modelling really comes down to the reasoning and understanding of the DBAs involved.

Howard stressed how important it is to focus on abstraction for specification. A quick Google search backs this up too: abstraction enables modular programming by hiding implementation details.

Figure 1 Dictionary Definition: Abstraction

Clarifying the Definition of Abstract for Specification

Howard clarified that the discussion isn’t about hiding implementation details within coding modules. It’s really about different levels of specification: business requirements, functional specifications, and technical specifications, and he pointed to the V model to help make sense of the specifying process.

Before turning to a ChatGPT-generated definition, Howard wanted to start with the general definition of specification, which he took from the Oxford dictionary: precisely identifying something, or stating a precise requirement.

Specification can also mean a detailed description of the design and materials used to create something, particularly in the context of database design and implementation.

Howard also pointed to Merriam-Webster’s definition, a detailed and precise presentation of a plan or proposal, before moving on to how you’d actually determine the specifications for a project. His suggestion is to work through six key questions: purpose, functions, performers, who, what, when, where, and how.

Figure 2 Dictionary Definition: Abstraction for Specification

Understanding the Importance of Specifications in Writing

Specifications spell out the tasks to be undertaken or materials to be used in a project. Merriam-Webster essentially defines them as instructions on how to create or perform something, and how the equipment involved will be produced.

John Zachman categorised specifications into various levels, planning, management, and architect specifications among them. Presenting specifications in a suitably abstract way to executives matters a great deal, and visuals, icons, and colour all help with that comprehension.

Specification documents aren’t unlike regulations or standards documents in that sense, and it’s worth digging into the details without ever losing sight of the original specification.

Figure 3 Dictionary Definition: Specification

Visual Presentation and Data Modelling in Writing

The example of a ranch house with a porch and kitchen, drawn from an extract by Steve Hoberman, is a good illustration of why visual presentation matters in writing. From a data modelling perspective, a conceptual model sets precise definitions and provides a framework for how things will actually work. Zachman’s data architecture factors in data alongside business processes and organisational structure to work out how and why things happen, and the Zachman ontology can define policies, procedures, and other aspects of business operations. Building a comprehensive business glossary comes down to defining it clearly, building a data model, and establishing relationships through a data dictionary.

Understanding the Definition of Specification and Abstraction for Specification

Identifying something and describing it in detail is central to specification. What specification is really aiming for is stating the requirements by eliciting core business concepts precisely, with a conceptual model representing those requirements and a logical model representing the business solution. Abstraction plays a key role here too, representing only the essential aspects of a system at different levels and leaving unnecessary details out, which enables modular programming and helps business stakeholders actually understand the requirements.

Figure 4 Understanding the Definition of Specification and Abstraction for Specification

Process for Representing Essential Aspects in System Modelling

Representing or identifying the most important aspects of a system means presenting it at different levels and leaving out unnecessary detail, especially when the audience is executives. That means defining the system’s objects and their relationships and building a model to capture the essential elements, then specifying the behaviour to sharpen that model further. It’s this process that lets developers create a more abstract representation of the system, which makes communicating with business or higher-level stakeholders considerably easier.

Abstract specification matters for understanding and adapting to changes in business and implementation requirements. When explaining the benefits of this to a business client, it’s worth emphasising the cost savings and the advantages of a more efficient, tailored system implementation.

The Six Friends of Writing Business Definitions

The V model, shown in the example below, is a genuinely useful process for drilling down into different levels of detail while making sure requirements get interpreted appropriately for each audience.

In the “Six Friends of Writing Business Definitions” course, breaking down business definitions means creating and discussing them one sentence at a time, debating and confirming each statement with the business, and considering related terms or objects along the way.

A data model can be built from these defined terms, abstraction for specification, process, essential aspect, non-essential aspect, with the focus staying on essential aspects, since those are what determine system behaviour and need defining at different specification levels without unnecessary detail.

The Importance of Defining Business Terms and Building System Models

Building a business glossary and understanding how terms relate to each other matters a great deal for defining business terms well. A comprehensive graph of those relationships makes it much easier to see how each term fits into the system model.

Defining abstraction itself is one real challenge, since it involves reducing multiple terms down into one, but building relationships between terms is what brings the definition to life. Agreeing on the process for defining abstraction before building the data model matters a great deal.

The level of detail and how glossary terms get presented should be tailored to the audience, executives, managers, architects, or developers each need something a bit different. Abstraction for specification should also demonstrate the system’s behaviour and link out to other areas of the system model.

It’s important to avoid including excessive information in the model that the audience may not appreciate.

Figure 5 Abstraction for Specification Structured Definition

Abstraction for Specification in Software Engineering

In software engineering, the V model isn’t just used for data modelling, it’s also used for extracting details and requirements from the business and moving toward implementation. Each artefact along the way, from requirements analysis through to coding, needs a corresponding test level for validation and verification, and that constant mapping between artefacts and their tests is what makes sure business requirements actually get delivered and signed off. Simplifying and presenting the right level of detail to the right audience means moving up through the hierarchy of levels, and the V model itself covers requirement design, system design, architecture design, modular design, and unit testing.

Figure 6 V Model

Agile Development and Test-Driven Development

Agile development starts with a business need and a product vision, and follows a V-shaped model from there. The process itself covers defining product features, creating user stories, and breaking components down into functional tasks.

Test-driven development is a big part of agile, writing the tests before the code and testing continuously to confirm functionality holds up, with regression testing making sure new changes don’t break what already works.

Product owner testing confirms the product actually fits its intended use and purpose, and business case assessment matters here too, covering return on investment and benefits realisation to validate the project’s success.

Different areas tend to use their own terminology, client needs, functional specifications, system or component definition, and agile allows for both top-down and bottom-up approaches to delivering functionality.

Detailed design needs to be agreed before moving to the next level, which avoids a lot of challenges and confusion further down the line. Worth noting: test-driven development isn’t limited to software either, it works in mechanical development too.

Figure 7 Agile V Model

Importance of Differentiating Functional and Non-Functional Requirements in Writing

A product’s requirements split into two types: functional and non-functional. Functional requirements explain what the product does; non-functional requirements explain how it does it.

Displaying a customer’s last name as a clickable link to their account history, or letting users sort by account open date, are good examples of functional requirements, while requiring strong passwords or supporting up to 200 concurrent users are examples on the non-functional side.

Business requirements sit at a higher level still, goals like reducing incorrectly processed orders by 50% or increasing repeat orders by 10%.

As the conversation moves from business requirements through functional requirements to system specifications, the level of detail and understanding shifts along with it. During requirements gathering, an analyst might produce artefacts like context diagrams, use case models, and conceptual data models.

Figure 8 Specifications required for Development.

Analysis Artifacts and Data Life Cycle Management

A range of analysis artefacts get used during software development to define requirements and implement business needs. Data modellers and solution architects build these up over time: user interface models, business process models, prioritised requirements, business rule catalogues, prototypes, all with the focus on extracting conditions from the business rather than getting into coding and background implementation.

The data life cycle runs through several stages: planning, designing, creating, storing, using, enhancing, and eventually disposing of data. Each stage produces its own specifications, data architecture, conceptual model, logical model, physical model, database creation, and everyone involved in the process has specific specifications to follow throughout the software development life cycle (SDLC).

SDLC stages include feasibility analysis, project planning, and requirements analysis, and staying within the appropriate level of detail at each stage matters a great deal.

Figure 9 Requirements Definition

Understanding Abstraction and Implementing Processes

In a conversation between Howard and Gauchet, both touched on how much abstraction matters in specification and how important it is to tailor communication to different audiences. They also flagged the need to justify the time and effort that goes into implementing a methodology or framework, by pointing clearly to the business benefit. Howard stressed the need for sign-off at different levels of detail, while Gauchet weighed budget constraints when choosing which steps to prioritise for system integration and testing, balancing profitability against quality.

Notes on Agile Development and the Zachman Framework

Agile development isn’t about doing away with documentation or specifications entirely, worth being clear on that. It’s really about the speed of implementation and the level of detail involved.

The end goal of agile development is a minimal viable product, built through smaller iterations and more precise definitions. Testing thoroughly at every step and level of abstraction is what prevents the costly mistake of pushing a faulty product into production.

Agile development tends to spark two main debates: one around reducing time and leaning on a waterfall approach, the other around making sure the right people confirm the need and definition of what actually needs building.

Agile puts real emphasis on drilling into the details faster and more efficiently, and a system’s functionality and features are what support its overall adaptability to change. When you’re adding disparate systems and integrating them, something’s needed to act as a catalyst for that.

The discussion also worked through the Zachman framework, which covers different levels of detail: data, processes, interactions, people, networks, timing specifications, and motivation.

Figure 10 Data Architecture Framework Abstractions

Developing a System Logic and Process Model

Howard stressed the importance of developing the system logic and process model together, thoroughly, during the development process, and of checking and verifying both. He also pointed to the need for the data side to stay synchronised with the process model, working through their relationship, and asked about any disparities within the data model, wanting clarity on the specific type involved. Validating functional or characteristic disparities within the data model matters a lot, he said, and Paul agreed, noting the importance of mapping to the test and making sure every necessary business requirement gets covered.

Figure 11 Relationship between Data Model & Other Models in EA

Importance of Avoiding Unnecessary Details in Business Presentations

At the top level of the V model sits the business requirement, the foundation everything else builds on. Adding unnecessary detail when presenting risks confusing the audience, so the focus should stay on communicating the vital aspects of the system through abstraction and specification. Skip the excessive detail that tends to spark arguments and confusion, and you save time and get agreement on the core aspects of the system faster.

When it comes to data models, starting at the subject area or data estate level tends to work best. Relationships and cardinality should get addressed at the conceptual level to keep the system clearly understood, and every component in a dimensional model contributes to understanding both the business requirements and the solution. Data governance is worth addressing early, even before data sets come into the picture, and identifying critical data elements through use cases feeds directly into the business glossary and supports a smoother implementation.

Importance of defining critical data elements and controlling scope in the conceptual model of a project

Agreeing on the critical data elements before setting expectations for data quality matters a great deal, and defining the scope of work quickly helps clarify what deliverables are actually expected. At the conceptual model stage, abstraction is worth avoiding, since it can pull in entities outside the project’s scope and significantly increase the size of the implementation. The more abstract the conceptual artefact gets, the harder the project’s scope becomes to control, and keeping that scope tight is what makes for an effective project strategy.