Data Migration Framework - Book Launch with Krupesh Desai
Executive Summary
This webinar outlines key considerations in ‘Data Migration Framework: Removing the Tech Debt,’ focusing on its strategic implications for businesses. Krupesh Desai, the author, explores the consequences and challenges associated with migrating legacy systems, emphasising the necessity for well-structured data migration plans and the integration of change management practices.
According to Krupesh, the book highlights the importance of managing agile changes in target systems and discusses effective testing strategies to ensure data integrity during the migration process. Krupesh also addresses the need for data governance and a maturity assessment to guide successful migration efforts. Finally, the webinar touches on the evergreen principles of project delivery and the significance of cutover planning while acknowledging the challenges posed by agile methodologies in this context.
Webinar Details:
Title: Data Migration Framework - Book Launch with Krupesh Desai
URL: https://youtu.be/n6SpaXyXp8o
Date: 21/05/2025
Presenter: Krupesh Desai
Meetup Group: DAMA SA User Group Meeting
Write-up Author: Howard Diesel
Contents
Data Migration and its Strategic Solutions with Krupesh Desai
The Implications of Data Migration in Business
The Consequences of Data Migration
The Challenges and Insights in Data Migration
The Process of Data Migration on a Legacy Systems
Implementing Data Migration in an Agile Environment
The Importance of Data Migration Plans
Data Migration and Change Management: Challenges
Data Management Maturity Assessment and Data Migration
Managing Agile Changes in Target Systems
Data Migration Testing Strategies
Effective Data Migration and Data Governance
A Discussion on Data Migration
Understanding Data Migration as a Business Transformation
Evergreen Principles in Data Migration Project Delivery
Cutover Planning and Challenges
Data Migration and Data Replication
Challenges with Agile Methodologies in Data Migration
Data Migration and its Strategic Solutions with Krupesh Desai
The book launch event featured Krupesh Desai, a seasoned CDMP Associate with over 15 years of experience in the data field. Krupesh highlights the release of his first book, Data Migration Framework: Removing the Tech Debt, on data migration, published by Techniques Publication on April 23rd and now available on Amazon. Additionally, he shared insights into his career journey, which began as a software engineer and evolved into roles such as data migration lead and integration manager in the public sector, ultimately leading to his current position as a data governance specialist in Melbourne.
Figure 1 Data Migration Framework: Removing the Tech Debt
Figure 2 Book Cover
Figure 3 "My Background"
The Implications of Data Migration in Business
Data Migration Framework: Removing the Tech Debt is designed to benefit ETL developers, technical professionals, and primarily project and program managers, as well as technical leaders involved in business system changes and data migrations, particularly those dealing with legacy systems. Krupesh emphasises the importance of viewing data migration not merely as a technical task but as part of a larger business challenge driven by significant changes such as digital transformations or business mergers.
As legacy systems often hinder innovation due to their deteriorated state, understanding this context is crucial for successful data migration. This involves transferring not just data but also the knowledge and meaning contained within it. The book's foundation is built upon three pivotal questions that frame this complex issue.
Figure 4 Key Takeaway
Figure 5 "Who is this for?"
Figure 6 "The Why?”
The Consequences of Data Migration
In 2022, after successfully completing his third project, Krupesh transitioned into independent consulting and focused on marketing myself through a white paper. During this process, he encountered a senior executive grappling with the challenge of retiring a critical 25-year-old asset management system at a municipal council, raising important questions about how to proactively approach this change despite having no clear target system. This experience inspired Krupesh to conduct extensive research over three years, ultimately leading to the realisation that he had enough material for a book rather than just a white paper.
Data migration, a crucial aspect of business continuity and customer experience, can have devastating consequences when mishandled, as illustrated by a case study involving a major bank that spent £318 million on a customer migration project, resulting in significant operational failures, financial penalties, and customer losses. This highlights the critical importance of properly managing data migration within digital transformation initiatives.
Figure 7 Three Profound Questions
Figure 8 "Why Data Migration Matters?”
Figure 9 Case Study of Data Migration Failure
The Challenges and Insights in Data Migration
Research highlights significant challenges in data migration, particularly from surveys conducted by industry experts. A 2017 study by Dylan Jones found that only 36% of data migration projects stayed within budget, and 46% were completed on time. In a more recent 2022 survey by Bridgehead, over 50% of CIOs managing healthcare data systems reported difficulties with legacy systems, citing the challenge of data migration as a key factor.
This complexity arises from the unknown nature of legacy systems, unreliable data dictionaries, and the ever-evolving landscape of target systems, which complicate the migration process. Additionally, such migrations are infrequent, typically occurring once every decade or two, which amplifies the risks of long-term inefficiencies and data management issues when mistakes are made.
Figure 10 Extract from Dylan Jones
Figure 11 BridgeHead - Hospital CIO Survey 2022
Figure 12 "What is unique about it?"
The Process of Data Migration on a Legacy Systems
In response to an attendee's inquiry about data migration from a legacy system, Krupesh clarified that the target system dictates the data structure due to new business processes and improvements necessary for the future. While the primary goal is to move data from the legacy system to the target system, there may be unique data elements or processes in the legacy system that the target system does not accommodate. In such cases, adjustments to the target structure may be required to incorporate these unique instances. Overall, the data migration process revolves around reaching the predefined target system.
Figure 13 "What is unique about it?" Pt.2
Implementing Data Migration in an Agile Environment
In an agile environment, ensuring a successful data migration requires establishing foundational elements in the target data model before planning the migration process. Data migration is complex due to the challenges of unknown sources and changing targets, necessitating involvement from business analysts and other stakeholders to map and finalise business processes beforehand.
This complexity is compounded by the need for ownership across various business units, as legacy systems often lead to inconsistent practices that must be addressed to prevent poor data quality during migration. Ultimately, a structured approach is essential to manage technical debt and facilitate effective migration while maintaining data integrity.
The framework presented addresses the complex issue of data migration, emphasising that successful transitions require more than just technical fixes; they also involve navigating interpersonal dynamics within a business. The process is divided into three stages: foundation, iteration, and migration.
The foundation stage focuses on project discovery and includes three parallel components essential for preparing before any code is written. After establishing a solid foundation, the iterative phase allows for the nuanced movement of data, culminating in the migration stage, which comprises three additional key components.
Krupesh then refers to his book, which contains 12 chapters, 10 of which are dedicated to exploring these components in depth, providing insights, and addressing various related questions based on three years of research and practical migration experiences.
Figure 14 "What is unique about it?" Pt.3
Figure 15 "What is the solution?"
Figure 16 Foundation, Iteration and Migration
The Importance of Data Migration Plans
‘Data Migration Framework: Removing the Tech Debt’ emphasises the importance of a comprehensive data migration plan as a vital document that drives project success, beyond merely serving as a courtesy artefact. Krupesh details how this plan can be leveraged as a key communication channel to enhance visibility and document decisions throughout the data migration process. Additionally, he encourages not to begin coding for migration until the foundational components are established and documented in the initial version of the plan, which should be validated by stakeholders.
This approach mirrors planning a holiday, aiming for a smooth experience while avoiding unpleasant surprises. Additionally, Krupesh notes that the framework discussed in the book addresses compliance and regulatory considerations, underscoring the plan's role in reflecting findings and engaging stakeholders consistently.
Figure 17 Highlights
Figure 18 Data Migration Plan - Key Document
Data Migration and Change Management: Challenges
Engaging in change management during data migration addresses resistance from individuals concerned about job security and legacy systems. Krupesh emphasises the importance of data segments, which define the logical partitioning of legacy data into master and transaction data, as well as the regulatory and compliance considerations that must be adhered to during migration.
The proposed framework outlines a proactive approach to data migration. It focuses on three foundational components before selecting a target system, ensuring that both internal policies and external regulations are incorporated into the migration plan through appropriate controls and vendor agreements.
Figure 19 Mind the Data Compliance
Data Management Maturity Assessment and Data Migration
Krupesh highlights the importance of data compliance during migration, particularly through a data management maturity assessment. This is crucial when dealing with legacy systems and individuals who may resist change, having worked for decades in established environments.
The assessment can follow two approaches: a technology-led approach, which involves leveraging an in-house data management team or hiring specialists if there is strong data governance maturity, and a business-led approach where business leaders take charge if the organisation's data governance is lacking. Both strategies aim to address the unique challenges of data migration while ensuring stakeholder involvement and support.
Figure 20 Data Management Maturity Assessment
Managing Agile Changes in Target Systems
Krupesh reflects on his initial project, where he had to develop a systematic approach to tracking the quality and progress of data structures during migration. Krupesh maintained four key flags for each field—indicating if it met requirements, had business rules, or had necessary data quality constraints—ultimately ensuring the data was loaded correctly into the target system. For straightforward fields like first names, where no business rules applied, I simply set the flag to 1, allowing me to gauge the completion of each data structure.
This method culminated in a dashboard presented to the steering committee every fortnight. Krupesh highlights the status of work completed and addresses any outstanding issues, particularly around delays in business rule approvals. Despite technical solutions being possible, the lengthy approval process for higher hierarchical structures, such as hospital ward codes, often hinders progress, demonstrating that challenges in data migration frequently stem from non-technical business processes.
Figure 21 Measure Data Migration Progress
Data Migration Testing Strategies
‘Data Migration Framework: Removing the Tech Debt’ outlines a data migration testing strategy that encompasses three key types of testing: data unit testing, data integrity testing, and functional testing. Data unit testing involves comparing the number of expected records, such as patients or customers, to those actually loaded into the target system while addressing data cleansing from legacy systems.
Data integrity testing ensures that the relationships between data points remain intact, preventing issues like customers not being loaded before their associated orders. Finally, functional testing requires manual verification of migrated data in the target system, and the book suggests employing statistical sampling methods to determine the appropriate number of records to test, ensuring a thorough verification process of data migration.
Figure 22 Data Migration Testing
Effective Data Migration and Data Governance
Krupesh emphasises the importance of a systematic approach to data migration, concluding each chapter with a set of reflective questions tailored for the reader's specific project. The second chapter, for instance, focuses on the scope of data migration and includes practical questions to guide project planning. By thoroughly addressing these questions from the first three components, readers can effectively complete their project discovery work.
The concluding chapter of Krupesh’s book highlights the significant investment involved in such projects, often exceeding hundreds of millions, and underscores the importance of aligning data governance and maturity with the data migration process. A strong data governance framework can facilitate smoother migrations while also uncovering gaps that present opportunities for establishing more robust governance practices. The book also aims to clarify key concepts such as data governance, data-driven culture, and treating data as an asset.
Figure 23 "Action Items for Readers"
Figure 24 Foundational Component - Scope
Figure 25 Concluding Chapter
A Discussion on Data Migration
The book's last chapter focuses on simplifying jargon used in the data industry and highlights the value of data migration as a strategic exercise for leaders. It emphasises that while project managers can benefit from the first eleven chapters, the final chapter is specifically designed for those who can influence the broader picture.
Krupesh expresses gratitude for receiving their first review after three years of research, which provided reassurance about the book's content. Questions arose regarding the role of subject matter experts (SMEs) in functional testing and the necessity of migrating all data before testing. Krupesh responded by noting that while SMEs should be involved, they are often limited in number, making it essential to engage super users and trainers for the comprehensive testing of larger data sets.
When managing data migration, particularly involving sensitive personal information, it's crucial to ensure that the right personnel are authorised to handle this data and oversee the process. Experienced teams, like SMES, can effectively navigate complex cases, drawing from their knowledge of unique customer or patient situations.
Krupesh shares an instance when performing a migration of 10,000 patients, it was only necessary to compare a sample of 384, which can be scaled when dealing with larger datasets. Effective training for data entry teams and careful consideration of compliance are essential for maintaining data integrity. Additionally, resources such as Brenner's shared link on common data migration mistakes can provide valuable insights for a successful migration project.
Figure 26 First Review
Figure 27 Closing Regards
Understanding Data Migration as a Business Transformation
An attendee posed a compelling question regarding insights gained from three years of developing a data migration framework. The key takeaway was the realisation that data migration should be viewed as a business transformation rather than merely a technical task.
Krupesh, initially a technical expert, shared how he came to understand the challenges faced by project managers and the importance of engaging with change management and business stakeholders. This shift in perspective highlighted the need for collaboration in addressing data quality issues, reinforcing that these are business problems that require ownership and accountability. Additionally, extensive research and case studies in Krupesh’s work further reinforced this view, emphasising the multifaceted reasons behind project failures and the necessity of communicating insights effectively within a business context.
Evergreen Principles in Data Migration Project Delivery
In a discussion on foundational components of data management, three key areas were identified as essential and Evergreen. First, compliance with laws and regulations is non-negotiable during data migration. Second, a thorough discovery and project planning stage is critical for ensuring clarity and direction throughout the project.
Lastly, having a structured data management plan is crucial; it allows for informed decision-making and control, unlike ad hoc decisions that often lead to chaos, especially when faced with constraints like budget and time. These elements serve as the fundamental foundation for successful project delivery.
Cutover Planning and Challenges
Krupesh highlights the critical importance of cutover planning in data migration, particularly for ERP projects transitioning data from legacy systems. He mentions that the book includes a dedicated chapter on go-live strategies, which addresses the challenges and best practices associated with cutover planning.
Key points include the impact of data migration scope on cutover duration, emphasising that larger data volumes lead to longer cutover times, potentially affecting business operations, especially in sectors like healthcare. Krupesh shares that his book outlines essential components such as the rollback strategy and includes reflective questions at the end of chapters to help readers consider their own cutover planning scenarios and decisions.
Data Migration and Data Replication
Krupesh highlights the differences and similarities between data migration and data replication, particularly in the context of moving data from on-premises to the cloud. While foundational elements such as understanding data volume, compliance, and technology remain consistent across both projects, key differences arise in data structure.
Data replication often involves transferring data without altering its structure, using methods like CSV dumps. In contrast, data migration necessitates adapting to a new system's requirements, which may include mandatory data points not present in legacy systems. This adaptation involves business rules and decisions to ensure data integrity and quality in the target system. Additionally, Krupesh touches on methodologies for project execution, emphasising considerations for the Big Bang versus parallel run strategies, as well as the challenges faced when employing agile methodologies in data migration projects.
When considering data migration strategies, organisations often face the choice between a Big Bang approach and a phased approach, each with its own risks and mitigation strategies. The Big Bang method involves migrating all data at once, making it crucial to define the migration scope carefully, such as focusing on the last three to ten years of data based on archival and retention policies. This method requires comprehensive change management and training for staff to ensure business continuity post-migration.
In contrast, a phased approach minimises risk by allowing organisations to learn and adapt through incremental phases, though it necessitates managing multiple systems in parallel, which can complicate data reconciliation and integration. Both strategies come with specific challenges, and a careful evaluation of legacy systems, change capacity, and data management dependencies is essential for successful implementation.
Challenges with Agile Methodologies in Data Migration
In a discussion about the challenges of using agile methodologies for data migration, Krupesh highlights his experience as a data migration lead, where agile practices were enforced rather than chosen. Initially, the process was painful, but the speaker adapted by developing templates and tracking progress effectively. Over time, Krupesh managed to navigate the complexities of data migration and manage stakeholder expectations, even as the number of data templates fluctuated.
By the final quarter, Krupesh notes that he was able to pinpoint specific issues related to data quality and mapping approvals, demonstrating that agile methods provided a vital structure for managing ongoing changes in a dynamic environment. Krupesh then concludes with a mention of contrasting approaches, noting that in waterfall-based ERP implementations, a different methodology might apply.
If you would like to join the discussion, please visit our community platform, the Data Professional Expedition.
Additionally, if you would like to be a guest speaker on a future webinar, kindly contact Debbie (social@modelwaresystems.com)
Don’t forget to join our exciting LinkedIn and Meetup data communities not to miss out!