Confession of a Program Manager (Part IV):Dress Rehearsal and Go-Live

Confession of a program Manager:

From the previous blog series you will remember the project phases, we had seen a sequence of

Inception -> Elaboration -> Install & Testing -> Dress Rehearsal & Go-Live -> BAU Stabilization

After the testing completion and obtaining the necessary change management board sign-off and getting infrastructure changes ready, it’s time for Dress Rehearsal and Go-Live.

According to my past experiences this is the most treacherous time of the project. By this time some of the team/(s) and team members are already near exhaustion and break point. One such example is of data conversion migration work stream. Before the testing (UAT / OAT etc..) is near completion,  the data conversion issues must have been ironed out to perfection. PM needs to ensure such things are properly managed.

As the famous quote goes “Failure isn’t fatal but failure to change is fatal!”

Encountering challenges during the lifecycle of the project are normal and the way we react to address it is crucial. Adapting right to change should be central theme to make it a successful implementation.

For the banking & financial institutions, getting the testing done right is utmost important. It’s better to identify any possible pitfalls and potential failures. Know it upfront rather than discovering it once the system is Live, so that things can be addressed. With a known issue at hand, either you will be in a position to accept the risk and absorb the impact or find a solution to address it. In both these scenarios the impact is reduced to acceptable levels.

Dress rehearsal time is critical and all the involved teams must work in harmony to make it happen. Some of the important aspects to keep in check for dress rehearsal are as follows:

Infrastructure fine-tuning & Support team alignment:  With all the excitement and rush, project manager must not forget to check the infrastructure readiness.  Whether the TEST and PROD segments are properly defined and access controlled. Admin & user accesses are only granted to necessary department and groups. Getting setup right at once is extremely important from both security and operations perspective.  Data base Administrators and Operations team alignment becomes critical more than ever in the lifecycle of the project.

Setting up Project WAR Room: Call it old fashioned or whatever, I strongly believe in setting up a WAR room or COMMAND CENTRE. This will allow critical resources to be stationed inside this facility –  physically. It could be a meeting room blocked for the entire duration or a co-located space on one side of the office floor dedicated just for the project teams. I would also suggest to add DBA, Infra team member and an IT support person to this setup. This avoids unnecessary time spend on reaching out and coordinating across various locations. This also gives everyone a sense of urgency and faster execution.

Change management: Getting necessary Change committee and Audit team approval for deploying the codes to prod segment, following their set procedures and format is important too. From past experiences I think it’s my job as a project manager to give change board enough confidence and convince them with necessary inputs so that approval process is smooth. After all, they are the gatekeepers of production environment and wants to ensure stability and safety of the system.

Adhering to conversion timings and fine tuning during Dress Rehearsal: Getting the data conversion migration timing right to the minutes & seconds is essential. The learning and experience that one gathers at the time of migration project is enormous. There should be no last minutes surprises or waiting for files being transmitted or converted. This being really the last opportunity to get it right.

Checking BCP and Fallback Mechanism: It is essential that during the system implementation all the aspects of BCP and fallback are checked properly. It’s not just the technology fallback but natural calamity fallback to focus on ‘People’ aspect and their back-ups for Go-Live. Sometime back, while I was working from Philippines, on the day of Dress Rehearsal we faced an earthquake of 6.1 magnitude. Entire office building were evacuated, thankfully nothing happened but it disrupted precious 2-3 days from our project plan just before Go-Live! People who were staying nearby were able to come back and join quickly and tried best to handle all possible things in absence of others. Thanks to the very friendly & helpful team! Empathy with your team and personal connect are essential to sail through such tough times.

Cut-Over Timing: Every implementation is unique and challenges are different. Getting migration cut-over timings right upto exact minute level is critical for success. PM must know where the dependencies are, where the slack is, where the decision points are and what is the latest and safest to pull the plug in worst case scenario.

SW Roll-back procedures: Often the part that gets neglected is SW rollback procedures. I find it natural as teams are engrossed in focusing on project aspects to make it live, make it a success they tend to forget the worst case scenario. Off course everyone likes to be optimistic and wants to move only in forward direction; but as a program manager one must think of all mathematical possibly and have a plan in place. During one of my past projects around 7-8 years ago, we faced this situation that my team was running out of cut-over timing for implementing a change for new product launch and we were forced to roll-back our SW changes. PM and line management had to answer lot many questions during that time until it was successfully implemented in subsequent week. Performing the overall risk assessment and being brave enough to take that right decision needs experience backed with knowledge and abilities and hence the role of program/project manager is so much crucial.

3^rd Party Connections: Ensuring the connections are established and handshake messages are exchanged is a pre-requisite. Communication is the single largest factor during this stage with internal and external stakeholders. It has to be both spot on and timely. Internal satellite system connectivity is equally important and must be checked well within planned timing. A simple but very effective tip here is, maintaining a team-wise roster along with time by which each system must get connected. Keeping a bridge line open for all participating teams spread across locations helps to a large extent but must be coordinated and controlled centrally from project WAR room

Coffee Manager Role: Sometimes I have taken up the most essential 😉 the vending machine manager role, during this time period. Ensure every team member is getting enough supply of Tea, coffee is a simple gesture but works to maximum. J This allows teams to work freely but at a pace that you want to maintain for project implementation. Removing obstacles from moving forward is the sole task and responsibility of PM at that point in time. Other areas to be usually taken care are logistics and transportation for people. As some of them will need to stay overnight and complete the assigned critical tasks.

Management Reporting: Whiles all these things are happening keeping a track of project and periodic reporting to project sponsor and board members is utmost important. PM must provide enough information and evidence to make an informed decision on Go or No-Go meeting. All the risks must be assessed, all angles covered and mitigation plan in place if needed.

Night of Go-live: Typically most of the implementations happen during the early hours of morning when the business and transaction are expected to be lower. Meticulously following the conversion and implementation plan to the exact steps in required to make it a success. And then you cross that all important line item from your implementation plan and then there is no turning back but ensuring the newly implemented systems are up and running smoothly.

Validating the important areas like Authorizations, IVR, ATMs, Channels, POS, Merchants, customer care, Branch connectivity, Schemes routing, inbound / outgoing data feeds, etc. It gives a sense of accomplishment and makes those sleepless nights and efforts worthwhile. You will cherish the smiles on you and your team members faces for a long timeJ.

I feel proud and fortunate to carry out this extremely important role with so many excitements, multiple times. Trust me, it’s a unique experience every single time.

One must be extremely cautious & careful in managing such large scale implementation programs.  Feels like a Doctor during execution, as the project needs to be analyzed, examined and monitored closely. Upon go-live, feels like a Surgeon coming out of the operation room after successful complicated operation of system transformation.

What matters most, is the end result. A Successful Implementation and happy client, its priceless!