Executive Summary
This webinar shares a comprehensive overview of Metadata Management, focusing on the role of Data Managers in Metadata Management. Drew Kennedy explores the historical and modern context of Metadata, its application in social media and music streaming platforms, and the essential skills and experience required for Data Managers. Data Architecture and privacy, Data Governance and Stewardship, and the challenges and solutions in Metadata Management are also explored. Importantly, Drew addresses the distinction between Metadata Management and Data Dictionaries, Business Glossaries, and Data Catalogues, as well as the difference between Metadata and Master Data. Lastly, Drew and the attendees discuss Metadata Management in data lineage and its relevance in different industries, emphasising the importance of team building, relationships, and compliance in effective Metadata Management.
Webinar Details
Title: Metadata Management for Data Managers
Date: 10 October 2024
Presenter: Drew Kennedy
Meetup Group: Data Managers
Write-up Author: Howard Diesel
Contents
Planning and Implementing a Metadata Management Specialist Exam
Understanding the Role of Data Manager in Metadata Management
The Historical and Modern Context of Metadata
Metadata in Social Media and Music Streaming Platforms
Metadata in Modern Data Management
Essential Skills and Experience for a Data Manager
Skills and Practices Necessary for Metadata Management
Data Architecture and Privacy in Metadata Management
Understanding the Roles and Responsibilities of Data Managers
Data Governance and Management in Metadata Management
Data Stewardship in Metadata Management
Data Management and Metadata Management Challenges
Technology in Data Architecture
Data Marketing, Training, and Education in Business
Team Building in Data Management
Building Relationships: A Case Study of a Metadata Analyst
Data Governance and Compliance in Different Industries
Metadata Management Policies and Solutions
Understanding the Difference between Metadata Management, Data Dictionaries, Business Glossaries, and Data Catalogues
The Distinction between Metadata and Master Data
Metadata Management in Data Management
Metadata Management and Data Lineage in Business
Planning and Implementing a Metadata Management Specialist Exam
Drew Kennedy opens the webinar and mentions the progress made on building the Metadata Management Specialist exam. He shares that this session will be a test run for some of the content. Drew encourages everyone to ask questions.
Understanding the Role of Data Manager in Metadata Management
The Data Manager’s role for Metadata involves providing content, context, and structure information about data to make it easier to find, understand, use, and manage. This includes specific details such as the data source, purpose, quality, content, structure, format, stewardship, business rules, and classification. The Metadata program encompasses considerations such as data ownership, retention periods, usage, confidentiality, and compliance with naming standards, security, and privacy regulations. Deep diving into these areas is essential for a comprehensive understanding of Metadata Management.
The Historical and Modern Context of Metadata
The practice of Metadata dates to the 3rd century BC with the libraries of Alexandria and Egypt, where the chief librarian, Callimachus, organised and catalogued the contents of the libraries using a system called “a Pinakis.” This system included comprehensive coverage of every scroll in the library, subject-based classification, an index of authors, and detailed descriptions of the scrolls. In modern times, the Clementine Library in Prague, founded in 1857, houses around 20 million books with a comprehensive catalogue. Presently, the importance of Metadata extends to e-commerce platforms like Amazon and Temu, where comprehensive product Metadata is crucial for business success. A lack of comprehensive Metadata can deter potential customers.
Metadata in Social Media and Music Streaming Platforms
We encounter experience Metadata in various areas of our lives every day. Social media platforms like Facebook and Instagram provide Metadata such as posting date, author, location, hashtags, comments, and likes. Statistical data for countries like Stats SA in South Africa also offers comprehensive descriptive, structural, administrative, and support information. Music streaming platforms like Spotify and Apple Music also utilise Metadata for artist, album title, song, genre, and lyrics. Similarly, platforms like Netflix and YouTube use Metadata to categorise and organise content.
Metadata in Modern Data Management
Metadata is not just a component but the backbone of modern Data Management. It is instrumental in organising and understanding data across various domains and industries. Its ability to identify and distinguish different data elements within an organisation makes it easier to search, browse, and retrieve information from large data lakes and diverse data repositories. Additionally, Metadata provides context and additional information about the data, aiding in proper storage, access, maintenance, categorisation for security and privacy, and sharing across organisations. It also underpins Governance, Reference Data, Architecture, modelling, Data Quality, security, warehouse, and analytics. However, managing Metadata is a complex and extensive task that requires broad expertise and significant responsibility.
Essential Skills and Experience for a Data Manager
For a Data Manager, expertise in Metadata Management is not just a requirement but a necessity. This includes developing and implementing Metadata standards, policies, procedures, strategy, and road map. It also demands a deep understanding of Data Management, Data Analysis, Data Engineering, Data Architecture, and Data Modelling. Data Governance skills and industry knowledge of the domain or industry where the organisation operates, including understanding industry-specific jargon, are also important. Connecting with business stakeholders and understanding their needs through experience in business intelligence and data analysis is beneficial for delivering substantial value.
Skills and Practices Necessary for Metadata Management
For a Data Manager to excel in Metadata Management, it is crucial to possess CDMP fundamentals, governance, and specialist-level knowledge of Metadata. Additionally, Drew recommends that strong interpersonal skills be essential for effective communication with stakeholders at all levels, including data owners, IT teams, and C-level executives. Leadership skills are also vital, as Data Management practices are often in their infancy and require guidance and motivation. Analytical skills, including problem-solving and identifying patterns in complex data, are equally important. Furthermore, expertise in classification systems such as taxonomies, controlled vocabularies, and proficiency in Excel and SQL can greatly enhance Data Management capabilities.
Data Architecture and Privacy in Metadata Management
Drew discusses the challenges of defining the search for Metadata and structuring catalogues to facilitate data discovery. Understanding different user expectations and including thorough tagging for effective data organisation is important. Drew expresses the necessity of classifying data for privacy, confidentiality, and retention, highlighting the legal obligations and the need for clear specifications in business requirements. He also mentions the intention to seek more information on data classification and the commitment to incorporating quality requirements into the Data Catalogue.
Understanding the Roles and Responsibilities of Data Managers
The responsibilities of a Data Manager include defining requirements, identifying stakeholders, and confirming data domains and critical data elements. Drew notes that it is important to understand the organisation’s drivers, whether they are legal requirements, access and retrieval needs, or others. Once the stakeholders and their needs are identified, developing a Metadata strategy is crucial. This involves a significant amount of work but can lead to quick wins by identifying critical data elements across different lines of business.
Data Governance and Management in Metadata Management
Drew emphasises the importance of defining Metadata standards and ensuring the quality of collected Metadata. Collaborating with the Data Governance team is imperative to establish and enforce Governance principles and policies. Addressing the importance of managing the Metadata lifecycle ensures data privacy and security compliance and supports reports on Data Quality, compliance, and Metadata usage. Drew underscores the role of Data Stewards, Data Owners, and Business Analysts in capturing and updating business terms in the Business Glossary and the Data Catalogue. He then highlights the potential for quick development of these processes using Metadata tools with strong reporting capabilities and structured databases.
Data Stewardship in Metadata Management
The challenge with Metadata lies in merging technical and business sources, such as data models, dictionaries, and quality Metadata, into a cohesive ownership model. While Data Stewards will be involved in managing glossaries and business Metadata, it’s essential to recognise the technical sources contributing to the overall Metadata landscape. Business collaboration is crucial in leveraging critical data elements and terminology to build a comprehensive glossary, linking it to physical Data Dictionaries through AI tools. This collaborative approach ensures that business terminology and technical Metadata are effectively integrated to drive the Data Catalogue.
Drew discusses the complex nature of Metadata Management within a business context. He emphasises the need for clear ownership and stewardship of different data elements. Drew shares the challenges of ensuring buy-in and commitment to capturing Metadata and the wide-ranging responsibilities of various data owners and stewards across different departments. A conversation then delves into the intricacies of Metadata requirements, stakeholder involvement, and the interconnected nature of data ownership and Management, using an example of a furniture retailer’s customer data to illustrate the diverse responsibilities distributed among multiple teams.
Data Management and Metadata Management Challenges
Many organisations are facing challenges with managing and utilising streaming data effectively. As more and more data is being streamed into databases without proper control, there is a lack of understanding of its contents. This leads to a repetitive process of adding attributes and creating data products without proper documentation. It is crucial to catalogue these data products in a Metadata repository to keep track of their origins, creation process, and utilised data. Capturing this Metadata requires the involvement of individuals in the Data Sciences team or data Engineers. Defining clear standards for different types of Metadata collection is essential. Still, it’s also important to prioritise efforts to ensure the best return on investment, as embarking on large-scale Data Management programs must result in delivering valuable outcomes.
Technology in Data Architecture
The Metadata Architecture is crucial and should be defined based on the chosen product, often influenced by other purchases like quality or modelling tools. Different types of Metadata architectures, such as centralised, distributed, or bi-directional, can be utilised based on ease of use and data-sourcing breadth. In addition to sourcing data, involvement in implementing the technology tool and designing the data model is important. This includes data structure, content, quality, physical databases, dictionaries, ETL, integration, warehouses, data sharing agreements, data usage agreements, and business process diagrams.
Data Marketing, Training, and Education in Business
A data professional’s responsibilities in managing Metadata programs include overseeing marketing, training, and education, ensuring correct training is available for different data users, promoting communication on goals and roadmaps, and celebrating wins. Additionally, there are opportunities to run awareness programs and roadshows to communicate the benefits of Metadata within the business community. Utilising a Business Glossary can streamline processes such as new employee onboarding and documentation creation while adding Metadata properties to entities, enhancing data accessibility. Overall, focusing on Metadata Management can yield significant value for an organisation.
Team Building in Data Management
When starting from scratch, it’s crucial to establish a team that includes Metadata analysts responsible for day-to-day tasks such as creating, maintaining, and updating Metadata records to ensure accuracy and consistency. These analysts work closely with stewards and domain experts who guide definitions, relationships, usage, quality, and compliance. Additionally, Engineers may be involved in maintaining technical infrastructure, creating code to integrate Metadata with various systems and APIs, and onboarding large batches of Metadata from diverse sources.
Building Relationships: A Case Study of a Metadata Analyst
Establishing strong relationships with various teams within the organisation is important for managing Metadata and ensuring Data Quality effectively. This includes working with Data Stewards, Business Analysts, Data Architects, Data Science and Engineering teams, Data Governance teams, and Chief Data Officers. Building these connections allows for better collaboration, understanding of business needs, and development of Metadata standards to support different areas of the organisation. Effective communication and collaboration are essential for successfully managing Metadata across the organisation.
Data Governance and Compliance in Different Industries
Drew discusses the importance of Data Governance and compliance, sharing his experiences in financial institutions. He highlights the requirements outlined in BCBS 339 for data lineage, Metadata, and ownership, emphasising the need for verifiable data lineage for compliance. Drews stresses understanding the drivers behind data accessibility, integration, privacy, evaluation, and monetisation. Additionally, he suggests considerations for utilising standards such as ISO, the Dublin Core Metadata Initiative, and geographic data standards based on specific business needs and systems. Overall, assessing and understanding Metadata requirements based on the intended use and lines of business is imperative before deciding on the appropriate approach.
Metadata Management Policies and Solutions
A discussion starts on the importance of implementing a comprehensive Metadata policy for Data Management. Drew discusses the need to capture Metadata according to standards, including classification for Data Quality, privacy, confidentiality, and retention. He also mentions the importance of linking the Data Dictionary to the Data Catalogue and highlighted the significance of data inputs, activities, and outputs. Additionally, there is a question about commercial products offering Metadata solutions. Drew suggests open-source tools like Data Hub and insights from organisations like LinkedIn and IBM.
Understanding the Difference between Metadata Management, Data Dictionaries, Business Glossaries, and Data Catalogues
An attendee expresses confusion about the differences between Metadata Management, Data Dictionaries, Business Glossaries, and Data Catalogues. Drew attempts to clarify by stating that classes, catalogues, and Data Dictionaries are all parts of Metadata, with the Business Glossary serving as the starting point for defining and capturing business terms. He suggests capturing detailed descriptions and linking metrics to data attributes in the Business Glossary or Data Catalogue. Drew also highlights the confusion caused by suppliers using different meanings for the same concepts, leading to a need for clear differentiation and definition of these terms.
The Distinction between Metadata and Master Data
Howard takes over from Drew and shares his understanding of the distinction between Metadata and Master Data. He demonstrates how the Business Glossary links to the Data Catalogue, Data Dictionary, and Data Quality rules compared to Master Data. Additionally, Drew highlights the unified meta-model, which aims to interconnect these elements, allowing users to navigate seamlessly through the glossary to locate specific data points or elements.
Metadata Management in Data Management
The discussion then moves on to delve into the complexity of managing Metadata, with attendees interested in the challenges of integrating, governing, and ensuring the quality of Metadata. It touches upon concepts such as meta-models and meta-meta-models and the vision of making Metadata visible and easily accessible to data consumers. An attendee delves into the idea of a data marketplace and the need for comprehensive lineage tracking. With the evolving nature of Metadata Management, Metadata is more readily available to those who need it.
Metadata Management and Data Lineage in Business
Howard shares the importance of human involvement in Data Management and the limitations of technology in understanding data lineage. He mentions the work of Irina Steenbeck, who authored a book on business data lineage and has developed a comprehensive approach to visualising data lineage, including horizontal and vertical lineage, impact analysis, and regulatory considerations. Her website, Data Crossroads, provides valuable insights into understanding data movement requirements, terminology, and creating metamodels. Thus, a holistic approach is necessary to understanding data lineage and integrating various data models, business processes, and Architecture.
If you would like to join the discussion, please visit our community platform, the Data Professional Expedition.
Additionally, if you would like to be a guest speaker on a future webinar, kindly contact Debbie (social@modelwaresystems.com)
Don’t forget to join our exciting LinkedIn and Meetup data communities not to miss out!
