Data migration is a complicated process, requiring dedication and expertise to successfully navigate the technical aspects. Without clean, transformed data, an ERP implementation will be doomed to fail before it really even begins. In this episode of The ERP Advisor, ERP experts, Shawn Windle and Quentin DeWitt, took a deep dive into the nuances of data migration, discussing how businesses can ensure the success of their projects by applying data migration best practices.
Data migration is the act of taking data out of one or more systems, transforming it from the format of the old system into the format of a new system, and moving it into that new system. That description sounds easy enough, but the process is fraught with the potential for mistakes and is the riskiest part of most ERP implementations.
A business may view it as a rather simple task of taking a customer and their data from the old, legacy ERP system and moving it into the new one, but at the end of the day, there are so many related pieces of information, so many ways of storing it, maybe in different systems from different eras, that shaping it from its old format into a new one can be a colossal task. Simply extracting all known data from multiple source systems, like a CRM, Accounting System, and proprietary database can overwhelm even the savviest of business users who are accustomed to creating their own reports from data extractions. The difference between creating routine reports versus identifying all data across multiple systems and grabbing all elements associated with something like a customer record are vastly different tasks that most clients underestimate at the outset. Teams who estimate the correct importance of data migration and assign enough resources at the beginning of implementation are ensuring their implementation can eventually launch into the semblance of a successful go-live where users see the correct data and will accept the system.
Some data may come from unstructured sources which complicates the extraction and validation process even more than data from databases. But the benefit of structured data for the new ERP significantly outweighs the effort to create and validate new data sets. The controlled processing within a new ERP ensures transactions must go through accounting, get checked for credit limits, and be monitored for credit limits rather than allowing loose processes which may invite poor business decisions or even fraud. This protects the company from costly mistakes and provides structure to report on the data from middle management to the executive team, all the way to the owners.
Data Migration is often considered the main cause of ERP implementation failure, likely because so many assume their team will move the data from point A to point B, with little technical assistance. Unfortunately, any implementation has multiple data types. Master records may consist of customers, vendors, employees, items, BOMs, projects, etc. Any one of these sets of records presents unique challenges in the cleaning, transformation, and eventual configuration requirements, and making mistakes in prepping it can derail the project. In our experience, many companies will tackle master records but will hit a brick wall once they confront how to migrate open transactions, trial balances, and, worst of all, historical data. Each of these data types will have unique considerations and difficulties which will frustrate the business users who are expected to prepare it. Check out EAG's data migration blog "Do I HAVE to Migrate Data into My New ERP?" for more information on data types and best practices.
At the end of an ERP selection, businesses must carefully review the implementation partner’s Statement of Work (SOW) to identify what the partner will do for them versus what the business will need to do for themselves. Most partners will provide their client with a template for the data structure that the client can follow at the beginning of the implementation and at key points during the project, the partner will load the newly transformed data for the client. However, the support normally stops there. Likely, the partner will only load the data into the system, and if there are errors, they will give it back to the client to fix. If a business does not have the resources to transform the data itself, it should plan on hiring data migration experts to facilitate its data transformation at the beginning of its implementation.
Many businesses neglect this due diligence because they don’t understand who really is expected to perform the data migration. Most ERP project leaders expect their new implementation partner to do it, but this is rarely the case. But when you check the SOW carefully, you will see the data migration effort is very limited to loading data only. By planning and budgeting for the resources for this important stream of work, the ERP project leaders will hedge the riskiest part of their implementation.
The Implementation Project Manager may realize the data migration is bogging down because the source data is messy. In the legacy system, there may have been a single item field, without a drop-down menu which led to employees manually entering information and creating incorrect item records. In these situations, businesses must put significant effort into cleaning inaccurate data and inactivating fields to prevent employees from classifying products in that way again. This could impact a single record or maybe tens of thousands, equating to a lot of work.
There are frequently duplicate records with similar names and inconsistent naming conventions. The users should take a major swing at cleaning the data they work with, inactivating records they will not need in the new system, and deleting erroneous records. Once data decisions are made it’s a smart idea to document them in a technical document.
Deciding whether or not to migrate historical transaction data is one of the most difficult aspects of data migration. Some ERP implementation leaders may be pressured by their users to migrate a large quantity of historical data or accounting general ledger without understanding the risk or impacts to the project. Most implementation teams have key people who expect to see years of historical information in their new system. What they do not realize is that the new ERP system does not need historical closed sales orders, closed purchase orders, closed projects, etc. in order to operate. However, they themselves as users WANT that information to do their work. This increases the magnitude of work which must be done to get data ready for the cutover and uses resources that would be better deployed to get the master records, open transactions, and trial balances ready. When querying users about their demand for historical data, assure them that legacy systems will still be available or will have “read-only” licenses for reference, especially for audits. The reason historical data migration can spell “doomsday” is that the team had already decided on parameters for master records, for instance, cutting off which items to migrate to the ones who have done business with the company for the last five years. But if the sales team wants to have closed sales orders, you may need to dramatically expand the set of items to archived or unavailable items so that you can complete all information needed for a closed sales order. Multiply that by the number of years, the number of closed transactions, or years of accounting general ledger data and you have a data migration nightmare that will be impossible to complete.
Data validation is the process of following data migration best practices to review, cleanse, and approve data from a previous system in order to migrate it into a new system. It is important to note that the implementation partner or the data team only shoulders the final leg of this responsibility – loading transformed data into the new system. Your team’s internal knowledge is vital to validate your data and confirm its accuracy and catch mistakes as early as possible. This is where Subject Matter Experts (SMEs) play a major role in data migration. SMEs help catch mistakes during data validation which highlights system configuration errors so they can be fixed well before go-live. SMEs will also be critical during the go-live weekend to do one final data validation. This vital step minimizes the risk of implementation failure and ensures the new system’s data will be usable and users can perform transactions.
In the end, data migration is vital to successful ERP implementation because the new system will be worthless without any data. Most ERP project leaders underestimate the time commitment it will take and become frustrated by long timelines and project delays. The key is for businesses to prepare for that time commitment. Any seasoned ERP champion would tell you, anytime there is a data change, leaders should expect it to take twice as long as they think it is going to.
The most important thing for businesses to do during data migration is to know what to expect. Once expectations are set, the business can prepare for the next steps and there will not be detrimental surprises. Need help? Contact a Data Migration Specialist with ERP Advisors Group, today!