Typical Challenges during Data Migration for Banks and Financial Institutes!!

“The biggest challenge of any data migration project is by far the data itself!” said the product expert. This statement triggered a series of questions in my mind related to the typical challenges faced by the team during a data migration project.

Is it really the ‘Data’ that is an issue? Or is it the ‘Understanding of the Data’ that creates the problem? Are there any ‘other reasons’ which may lead to data issues? I tried to find the answers to these questions during one such recent data migration project.

A data migration project typically follows following stages –

Data Analysis -> Data Mapping -> Data Extraction -> Data Transformation  ->Data Loading   -> Data Reconciliation

Each stage comes with its unique set of challenges and requires specific corrective actions to ensure a successful migration. This blog is focused around the issues and challenges faced during one such recent data migration project where a mid-sized finance company migrated their credit card portfolio from cards processing environment to an internally hosted licensing card management system solution. Both these card management solutions were industry standard off-the-shelf solutions with multiple customers across the globe. Hence the expectation from senior management was to breeze through the migration without any major hiccups.

But in reality, there were hiccups…. and that too plenty of them!

It all started when the finance company was trying to launch some innovative card products for its customers in the market but faced a lot of resistance from the third-party card processor as the new product features were not readily available in their processing system. Development of the new features would be an expensive proposition and also time intensive. The benefits they were getting from the processor while growing from a small-sized business to a mid-sized business had started to fade away. A decision was made to migrate to a modern system which could be hosted inhouse giving them complete control over the system.

After a detailed evaluation of multiple systems, this finance company decided to migrate to a new system with significant presence in their geography and also with a good support structure. The plan was to complete the migration within a time frame of 4 months as the product vendor had performed multiple migrations from the same processing system to their licensing system during their previous projects. They had readily available migration jobs which could be reused for this project.

I believe it’s easier said than done. Each migration comes with its own set of challenges which everyone involved realized it by the end of this project.

The team faced its first challenge during the data analysis stage. The business team did an analysis of the historic data on the screens to identify all the information that would be required from the old system to have a smooth transition to the new system without affecting any existing functionalities. Little did they know that the data in the screen didn’t match some of the data in the backend files. At that point, they didn’t have an expert in the team who understood how the old system was designed in the backend. The processor also sensed that the partnership with this finance company was coming to an end, so they might not have reviewed the requests thoroughly and shared the data without much further analysis. The quality of data was poor and as a result the first round of data migration was not at all satisfactory, leading to loss of time and energy for the whole team.

The data mapping and data transformation stage brought the next set of challenges for the team. The issue was identified for the Non-Performing Accounts NPA’s which were maintained in a different inhouse system to avoid paying the processing fees for NPAs. During the data mapping stage, it was observed that some of the mandatory data in the new system was not available in the old system for those NPA accounts. This included customer’s statement & transaction history. A lot of processing in the new system was dependent on this data. The only solution available to the team was to rebuild this history data with some assumptions. The data was rebuilt, but the efforts and time consumed for this exercise was huge. It also introduced a lot of additional issues which had to be handled during and after the migration.

The next challenge was completely unrelated to data but required a significant correction of data before migration. As per the original plan, the migration was scheduled before the start of the holiday season on a statement cycle so that the transactions posted in the old system could be cycled in the old system and there was no impact on interest processing. To achieve this, the entire portfolio was updated to have a common statement cycle on which the entire cut over could be done. The planning was good and all testing was performed based on the above understanding. Suddenly three weeks before the planned migration day, the processor informed that the migration could not be continued on the planned date as it coincided with a freeze period. After multiple rounds of discussions, the business agreed to migrate the portfolio 2 weeks after the planned migration date when the freeze period was over. They couldn’t wait till the next cycle date as that would have coincided with another freeze period and festival season. Thus, they ended up selecting a new migration date that was a non-cycle date. The industry accepted strategy for migrating on a non-cycle date was different from the strategy for the ones that migrate on a cycle date. Additional steps had to be introduced in the migration strategy for non-cycled accounts (which was now effectively the entire portfolio!). The new product vendor agreed to avoid these steps and make data corrections in their system during the migration to handle this change. A completely unrelated event had led to significant changes to the data which later caused other issues during the migration.

The last challenge I would like to mention here was related to the data reconciliation stage. One critical aspect of reconciliation was identified as Reconciliation of Balances. The balance reconciliation would ensure that there was no leakage of balances during the migration. According to the reconciliation strategy, it was decided to perform the reconciliation at each Product Level. The reconciliation went fine for couple of products but started giving major differences from the third product onwards. What had gone wrong all of a sudden from the third product? After some analysis it was identified that the products after the third product supported cards with split credit limit. It was also identified that some 10 years back these split limit cards were mysteriously changed to combined credit limit and then changed back to split limit cards. No one had an idea about why and how this exercise was performed as the old processing system did not support such a change. Without wasting further time, a decision was taken to correct such cards in the old system before migration. As part of balance reconciliation additional information was extracted to identify each account with the above issue and provided to the business for manual corrective action.

This brings us back to the question whether the biggest challenge of any data migration project is the data itself? Having this recent migration project experience fresh in my mind; I would like to conclude that data need not always be the biggest challenge of a migration project. A proper planning for the project, detailed analysis of the data and understanding of the source and destination systems could bring down the data issues by significant level. Correctness of the data is relative to the system that processes it. While migrating it proper care should be taken to make it relevant for the new system.

In summary all these elements matter a lot, meticulous planning, detailed design, deep understanding of source and destination system architecture, experience & expertise to tackle data exception during migration cut-over all go hand in hand and a long way. Reading about migration challenges and executing a migration is altogether a different experience. Getting the people with right skillset and expertise right from inception will help execute such complex and important projects for any financial banking institution.

In case you are thinking about embarking on one such excursion, Verinite is happy to help you succeed on your journey. For more information on our offered services in this space you can contact us by sending an e-mail to “info@verinite.com”