What are the unique challenges and considerations while conducting Performance Testing for Payment Switches / Applications?

INTRODUCTION

PERFORMANCE TESTING checks the speed, response time, reliability, resource usage, scalability of a software program under their expected workload. The purpose of Performance Testing is not to find functional defects but to eliminate performance bottlenecks in the software or device.

Performance testing is always been an integral part of information technology sector. It becomes even more essential when it comes to critical areas such as banking, stock market, e-commerce, online ticketing services and fintech. These are the areas wherein poor system performance can restrict potential revenue of the organization; ultimately causing real-time and non-retrievable – financial as well as reputational losses. A tiny issue in the system’s performance can lead to huge financial losses.

To avoid system performance related direct or indirect losses, organizations have become cautious about the health of their applications, systems and infrastructures. Over the period of a decade, the demand for performance testing have been increased exponentially. And because of higher demand, performance testing itself have become an individual domain. Result of which, a lot of relative tools and expertise have got introduced.

But there are still some areas, where performance testing is not popular. In our todays blog we will talk about one of those areas and we will try to understand what are the reasons and challenges behind it.

Here I am talking about Switching technology. As we already know that it processes any card-based or payment transactions initiated at various acquiring channels like ATM, POS, e-Com, IVRS, mobile/internet banking etc.
Switching technology is one of the important areas of FinTech and it’s been standardized and accepted globally. Some banks are having their own switching environment and some of them are going for a processing platform i.e. shared environment. But all in all, todays fact is – none of the bank can survive without switching technology. Day by day the number of switch-based transactions are increasing enormously. This switching domain can also be described as 2nd largest and critical areas in banking sector after core banking system.

In spite of being such an important entity, the performance testing for this domain is yet not grew as expected. The challenges are quite interesting to know.

 CHALLENGES AND CONSIDERATIONS

  1. Complexity in Variety: Switch technology is a combination of multiple service areas, and solution of each service area varies from another. It means a performance testing solution developed for one service area doesn’t remain suitable to another. So, each service area needs a distinct solution and because of this, the scope to develop a performance testing solution for switch becomes vast.E.g. One service area is debit card management system and another is transaction authorization. Both the service areas are equally essential but the solutions are totally different.

Likewise, regarding load testing of transaction authorization – a solution to create automated load of ATM transactions is different than that of POS or e-commerce transactions.
Similarly, to validate performance of EOD activities a different solution is required to create a load.
Hence there is no single tool available in the market which can be termed as a single standard solution to conduct performance testing for all the service areas of switch.

  1. Proprietary designs and their distinctiveness: Payment Switch is been a perfect competition market since long time. Many companies have launched their proprietary products and working hard to find creative technical solutions, in order to attract clients and stay-fit in the market.Considering this fact, we have a lot of Switch products available today which were developed to serve a common purpose of the industry. However, each of the product is distinct because of three facts – product architecture, solution and technology.

Even though all the companies have designed their products considering the norms of Payment Card Industry, internally each of the product is designed in its own distinct way.  Because of which a solution developed for one product cannot be replicated to another product from a different company.

E.g. Measuring a response time of any activity or transaction is one of the parameters in performance testing.
During load test suppose a delay is been observed. Then to identify the cause of that delay, certain check-points are applied internally on modules/procedures of the Switch. And this can be achieved with the help of expert solutions, by making some technical interventions inside the Switch system. These technical interventions become system specific to that particular product only, which cannot be replicated to another Switch of different make.

  1. Exclusive services lead to unique solutions:Each switch is unique in its own way! Even if two different banks from same country are having their switch of same brand, both the switches will be implemented with different solutions because of the dissimilar requirements.Banks have started focusing on how to stay updated with the market. Keeping customer satisfaction in mind, launching innovative ideas and taking experimental approach have become common.

Looking at the demand in EFT, cards and payment domain, banks are trying to provide USPs by availing products or services uniquely in some or other way.Result of which lot of customizations are happening locally to cater variety of requirements or enhancements.

Even if a same bank is having presence in multiple countries, they are required to maintain a different switch solution for each of the country. This is to cater the mandates applied by national and international interchanges. Here the mandates play the role of differentiator. Hence a common performance testing solution cannot be applicable even if multiple banks having same type of switch product.

  1. Product trust and system customizations: We saw above four direct challenges which occurs while conducting performance testing. But there is one indirect challenge which is also equally responsible. This indirect reason is nothing but the blind product trust which customers are having, due to the goodwill that product companies are having in market. This challenge is not the one which occurs while conducting performance testing but, it has definitely reduced the requirement of performance testing in overall market.To elaborate – During product finalization phase, bank validates if the necessary requirements are met or not. Product company also makes their customers satisfy by providing accurate solutions and value adds, by doing multiple customizations in standard product. These customizations can be at program, solution, module or architecture level.

But due to such numerous customizations the system becomes different than the standard product. It becomes more complex, which can cause impact to systems performance.
Studies have been reported many incidents so far, wherein a performance of final implemented product is observed as a degraded one. Here we need to understand that, to meet certain performance norms a system has to undergo some benchmarking processes. Benchmarking is always performed on standard product and not on customized versions of it.

And the problem statement is, after doing all the customizations banks still believes that the system will perform as per their requirement. However, during initial stages system may work as expected but as the time flies, number of system users, transactions per second and other system activities start increasing. Result of which, slowly and gradually performance related issues start impacting the system and ultimately to the business.

As per my understanding, any bank should conduct performance testing at least on below events –

  • Implementing a new switch
  • Migration
  • Version upgrade
  • Rise in transactions, activities or users
  • Any change at product architecture
  1. Absence of universal solution:Till today there is no single solution, which can automate the entire process and serve the purpose of end to end performance testing. However, it can be possible by using multiple tools to cater each of the service area of Switch.  For example, to create load on UI based applications, we can use Load runner, JMeter etc. But these applications cannot be used to generate transaction related load, and for which we have to use domain specific applications like FINSIM, NeaPay etc. which comes with hefty prices.

We have already discussed about the challenges to conduct performance testing in reason 1, 2 and 3. These challenge remains unchanged when it come to development of a new tool that can be a universal solution of for this domain. But it is also not impossible to develop such a solution. However, developing such solution is a huge scope looking at the demand of this niche market.

Considering this fact of demand v/s market structure, chances of arriving universal solution within near future are very minimal. But having said that, it doesn’t reduce the criticality of performance testing of Payment Switch application.

Before concluding, I would like to summarize this topic as follows–

  • Although a standard Switch product catalog defines the suitable system performance, it is still important to conduct Performance testing.
  • Conducting end-to-end performance testing for all the service areas of Switch is a challenging activity.
  • There is no standard product available in market so far, which can be used as universal solution.
  • It is very much possible to conduct performance testing to measure performance of all the service areas using combination of solutions.

Manual and Automation Testing Challenges

Software testing helps to eliminate the probability that the final product has errors of design or functionality.

Software companies perform product testing either manually or by using automation tools. It is an equally integral part of the software development lifecycle just as the design and development. As a software tester, I faced lot of challenges at beginning of my career like building automation script it used to be tough to integrate multiple applications to execute one functional scenario withing strict deadline. Software testing has many such challenges whether it is Manual or Automation and every tester would have experienced at least one or more of the challenges which we have listed down.

Challenges in Manual Testing

  1. Understanding all the requirements: When requirements are documented and provided to a tester, it is tester responsibility to understand each requirement properly and if there are some ambiguity then tester should talk to client and get it cleared. If tester fails to do so, he/she will not be able to test the software properly. Tester should try and find maximum defects to make sure final release is defect free. All the possible scenarios should be covered so no defects get missed.
  2. Regression test case coverage: After every development release or defect fix, we need to ensure that previously working functionalities are not adversely impacted. Tester needs to understand the functionalities that are fixed in new release and affected code/functionality to update regression test scenarios.
  3. Out of box thinking: It is not always possible to write all the scenarios, most of the time while execution tester needs to think out of box to test different scenarios which could be very rare or unique to the installation.
  4. Cope up with new/modified requirement: Sometimes requirements are changed/added in between the project milestones. It becomes a challenge to cope up with the new changes within the deadline. As we need to understand new/updated requirement, update test cases, regression test cases.

Challenges in Automation Testing

  1. 100% Automation: Do you think 100% automation is achievable? No, it is impossible to automate each and every test case, it is a challenging job to automate maximum number of scenarios possible. For example, many a times if there are image related scenarios like captcha entering while login this kind of scenarios could not be automated.
  2. Requires appropriate technical skill: When we talk about Automation, tester has to have proper programming knowledge to write the scripts and also should show how to use automation tools really well.
  3. Upgrading skill set as per client requirement: Many of the time it happens that manual and automation testing team / person is same; and based on client requirement tool may change, so tester needs to be always prepared for upgrading the Automation skills. Tester needs to understand the tool and have hands knowledge within a short span of time.
  4. Complex scripting (integration with multiple application): If the project timelines are tight and functionalities are complex it becomes very challenging for tester to write the test script, execute and verify the scripts. This is the time when tester has to show his proactiveness and smart work to complete the automation scripting within deadline.

In summary, one can overcome all these challenges by self-learning, upgrading skills, learning new things. Even though there are so many challenges in software test field but still it is one of the best jobs as it ensures delivery of quality product at the end.

 

Confession of a Program Manager (Part III) –First Install and Testing Phase

From the last blog if you remember the project phases, we had seen a sequence of

Inception -> Elaboration -> Install & Testing ->Dress Rehearsal & Go-Live ->BAU Stabilization

In this blog we will be talking about the First Install and Testing
As it is famously said that well begun is half done, but the real fight still awaits to make it a total success. In the previous two phases most of the action is happening either remotely or with select business, IT, Operations, vendor teams. But when it comes to first install and testing, it’s time for majority of the teams to get involved and get the ball rolling.
As a program manager some of the things that needs attention and monitoring are listed below:
1) First Install of the product: Once the product vendor is ready with their pieces of enhancements and customizations it’s time to install the SW and make it up and running for shakedown. Depending on the strategy approaches might differ for installation. Everything gets installed in one go or a phase wise / drop wise approach is taken. Infrastructure team and bank internal IT team gets in to action to check all connections, integrations, handshakes, setting up test and prod segments. Getting necessary approvals from Change Management Committee and in some cases handling unplanned challenges.
 2) Department Involvement: after the first install and systems are shaken down for confirming the basic connectivity and handshakes, it is good to be handed over for next steps. Typically it involves Training and system integration testing before the functional UAT starts.
It’s essential to get the order of activity sequence right. An experience project manager needs to ensure all the dependencies are well understood and factored in while coming up with sequence of activities. Although the challenges are always new and specific to that installation.
Some quick tips for aligning important activities
• Get departments aligned – for the set of activities they need to perform, get their commitments and structure in place. It’s a good practice to involve them in the working committee meetings and track progress
• No surprises – Sometimes we have observed in the past that due to various reason like change in department head or late realization some tasks gets added to the list and becomes mandatory for obtaining sign-off. Example late request of getting vulnerability testing done or adding another round of operations users testing on top of already planned activities. As a project manager it has to be wisely understood and quick decisions taken to make necessary amendments.
• Walkthrough sessions – Over the years as a project manager I have experienced that walkthrough session are far more effective than the number of emails sent to people. It’s time saving and you get instant buy-in and common understanding across teams during such sessions.
• Teams under one roof- getting dedicated team and co-location is equally important for faster interactions and quick progress. However the reality is most of the times, the teams are neither located at the same place nor same country / time zone, effective co-ordination and management becomes key.
3) Trainings: As the saying goes ‘Tell me and I forget, teach me and I may remember, involve me and I learn’. One of the important activities for business, technology and operations team during this time is to undergo hands-on system training, self-learning and increasing their expertise and knowledge on the new system being implemented.
From past experience couple of tips for project managers. Arrange the training only after the first SW install that is specific to bank requirements rather than any vanilla version of install.In order to save time is try and get the training for operations and technology, business done in parallel if possible as the content topics differ.

 4)Testing: This is one of the most crucial phase of the project. Testing must complete on time and successfully. The outcome drives the go-live and overall success or failure of the project. Depending on project size and complexities various types of testing are required like SIT, UAT, OAT, Interface, Regression, Performance, Security, Vulnerability and the list continues. The earlier defects are detected in the testing cycle, better it is.

As an industry best practice, it is advisable to carry out multiple rounds of testing specifically the UAT rounds. Typically 2-3 rounds of UAT based on the complexity and volume of the changes that are being introduced. This should be followed by at least 1 clean pass run of regression testing ensuring the last lot of changes that went in to the system are not impacting the previously working functionalities.

The number of UAT rounds is also dependent on the number of code drops to the environment and making a prudent decision specific to project instance will benefit most. Example – at an installation client had taken an approach of monthly code drops 4 in total, so UAT rounds needed to be extended to match with code drops and increased to 6 rounds in total.
An experienced project manager will definitely be able to sail through these decisions quickly and confidently.

Another important thing is the decision on how many open incidents should be acceptable before exiting the testing phase. In an ideal world this count should be Zero but it seldom happens. This decision is tricky but ensuring that no critical or high priority incidents are open is essential. There could be some medium or low priority incidents hanging around but then having a mitigation plan and knowing how to tackle it in production is important.
It’s all about finding the optimal release point rather than fixing all miniscule defects and loosing precious time resulting in further delays and overall costs to the project.

5) Payment Scheme certification: No matter how fast your testing team progresses and completes the testing the Payment schemes must certify in order to go-live. There are different elements of certification testing, Issuer Host certification, CPV, Acquirer certification. Every scheme has its own procedure of allocating their resources and opening up the project with banks. Experienced project manager helps greatly reducing the time wasted during initial information exchanges with schemes, with prior knowledge of what is expected. It is essential to factor in this critical timeline for overall completion.
After all these hurdles get near to completion, program manager must take fairly access the situation whether all the exit criteria are correctly met or not. Are there any exceptions? If so, why and with a plan to tackle them individually. Risk assessment for the open incidents what are acceptable levels and what is not etc. It takes lot of effort and energy to get all these things sorted and in order.
Things may not be perfect, never will be perfect. For me, what is essential at this point of time is having the pragmatic approach. Crux is to know the ground reality, what is possible in given time, what best can be done in order to close open points and proceed to the next stages.
As someone has said ‘Progress is more important than the perfection’ the decisions need to be taken at appropriate time and keep moving in right direction to achieve project goals.

What needs to be tested during CMS Migration Project? How and Why?

Card Management System (CMS) Migration:

The Card Management system migration projects is one of the critical projects for any institutions or bank processors; as it involves the data which is extremely critical to banks and data migration is challenging in nature. Data migration is a prominent data-movement technique that is usually combined with other techniques.

The CMS migration includes the migration from legacy system to newer system, which includes data and other internal & external systems like Cards, Collections, Dispute, Statements, Embossing, PIN Mailer, HSM & keys, Payment Scheme and Authorizations etc.

What is CMS Migration Testing?

The CMS Migration Testing is a verification process of migration of the legacy CMS to the new CMS with minimal disruption/downtime, with data integrity and no loss of data (Data correctness), while ensuring that all the specified functional and non-functional aspects of the CMS are met post-migration.

CMS Migration testing includes

  • Functional testing
    • Regression
    • Functional regression

The regression testing is to perform testing with old data, new data or combination of the both, old features (unchanged features of CMS), and the new features of CMS, if any. The functional regression testing is to perform testing with newer data as per test cases defined by the testing team.

  • Data Migration testing
    • Mock
    • Dress rehearsal

The mock testing is to perform testing with migrated data to perform data validation, data sampling and data reconciliation. The dress rehearsal testing is critical activity for CMS migration projects. This includes the migration from legacy to newer CMS along with data migration, HSM & key migration, other internal & external systems.

  • Non-Functional testing
    • Certification
    • Performance testing

The Certification testing is to perform and validate the payment scheme (VISA & MasterCard) set of pre-defined test cases. The Performance testing is to determine how a system performs in terms of responsiveness and stability.

Designing the test strategy for migration include a set of activities to be performed and few aspects to be considered. This is to minimize the errors and risks that occur because of migration and to perform the migration testing effectively.

  • Data sampling
  • Data validation
  • Data reconciliation

Critical Points for CMS migration:

Below are the critical points for CMS migration:

  1. Identify the functionality from legacy system to new system
  2. Prepare the network diagram
  3. Identify the list of new functionalities for newer system
  4. Approach for data migration
  5. Data mapping
  6. Data validation
  7. Data reconciliation
  8. Payment scheme, if any changes
  9. Embosser , if any changes
  10. Statement vendor, if any changes
  11. Interfaces (ESB / EAI API’s), if any changes
  12. Card Plastic, if any changes
  13. CMS end points (ICA)

Why CMS Migration Testing?

The CMS migration to a new system could be for various reasons, system consolidation, obsolete technology, optimization or any other reasons.

While the existing legacy system in use needs to be migrated to a new CMS, it is essential to ensure the below points:

  1. Any kind of disruption/inconvenience caused to the cardholders due to migration needs to be avoided/minimized.

Example: downtime, loss of data, transactions declines etc.

  1. Need to ensure if the cardholders can continue to use all the CMS features by causing minimal impact during migration.

Example: change in the functionality, removal of a particular functionality

In order to ensure a smooth migration of the CMS, it is essential to carry out Migration Testing.

Technically, it is also required to be executed for the below purposes:

  • To ensure compatibility of the new/upgraded application with all possible hardware and software that the legacy application supports. In addition, new compatibility should be tested for new hardware, software platform as well.
  • To ensure all the existing functionalities works as in the legacy application. There should be no change in the way how the application works when compared to the legacy one.
  • The possibility of a large number of defects due to migration is very high. Many of the defects will usually be related to data and hence these defects need to be identified & fixed during testing.
  • To ensure whether the System response time of the new/upgraded application is the same or less than what it takes to the legacy application.
  • To ensure if the connection between servers, hardware, software etc., are all intact and do not break while testing. Data flow between different components should not break under any condition.

Migration testing life cycle

  • Pre-Migration phase

Before migrating the data, set of testing activities are performed as a part of Pre-Migration test phase. The important activity is to create “migration strategy” document along with identified set of test scenarios, test plans and test cases.

  • Migration Phase

Migration Strategy needs to be strictly followed to carry out the migration activity. Ideally, the migration activity begins with the data back up and conversion, so that, any time the legacy system can be restored.

This activity is extremely important and need to perform Mock runs for data sampling, data validation and data reconciliation with both legacy and newer CMS.

In addition, EOD validation need to be check with legacy and newer system to validate the accuracy of transaction amounts and statements.

Migration Test Summary Report

The test summary report should be produced after completing the migration testing and should cover the report on the summary of the various tests/scenarios carried out as part of various phases of migration with the result status (pass/fail) and the test logs.

Following activities need to be recorded during migration:

  • Total time for CMS Migration
  • Downtime of the applications
  • Time spent to CMS migrate
  • Time spent for rollback

In addition to the above information, any observations /recommendations can also be reported.

  • Post-Migration Phase

During Post migration testing, perform out the functional testing with older and newer records in the CMS. In addition, data reconciliation and CMS end points with ESB middleware, Collections, Payment scheme’s and other internal system need to be checked. As this might have customer impact.

Role of Data Science and AI in Improving Test Automation!!

As the world is moving faster with emerging new technologies the need of improving Test automation also grew. Now when large data is required for testing within less time latest technologies such as Data science and AI can be helpful to improve Test Automation. Where data science can help in generating data from different sources AI can help in creating codeless automation based on data generated.

What is Data Science and AI?

Data Science is a blend of various tools, algorithms, and machine learning principles with the goal to discover hidden patterns from the raw data. Data Science is primarily used to make decisions and predictions making use of predictive causal analytics, prescriptive analytics (predictive plus decision science) and machine learning.

Artificial intelligence (AI) is an area of computer science that emphasizes the creation of intelligent machines that work and react like humans. Some of the activities computers with artificial intelligence are designed for include: Speech recognition. Learning, observation.

Need of Data Science
Traditionally, the data that we had was mostly structured and small in size, which could be analysed by using the simple BI tools. Unlike data in the traditional systems which was mostly structured, today most of the data is unstructured or semi-structured. This data is generated from different sources like financial logs, text files, multimedia forms, sensors, and instruments. Simple BI tools are not capable of processing this huge volume and variety of data. This is why we need more complex and advanced analytical tools and algorithms for processing, analysing and drawing meaningful insights out of it. Let’s dig deeper and see how Data Science is being used in Banking domains.

How about if you could understand the precise requirements of your customers from the existing data like the customer’s past browsing history, purchase history, age and income. No doubt you had all this data earlier too, but now with the vast amount and variety of data, you can train models more effectively and recommend the product to your customers with more precision. Wouldn’t it be amazing as it will bring more business to your organization?

Let’s take a different scenario to understand the role of Data Science for Fraud Detection. What if customer who always makes transaction less than 500 rupees per day and suddenly his transaction spikes to 10000 a day isn’t is suspicious. In such case Data science can help in early prediction and restrict the account from making such transactions . Based on customers previous transaction history, location history, income history and data from other sources like social media will help data science to predict and act against the fraud.

How Data science and AI will improve Test automation?
Traditional testing techniques still rely on humans to source and analyse data. But let’s just say that humans are not infallible and are quite prone to making poor assumptions.
The less time there is for handling data, the greater the chance that testing will produce skewed results with overlooked bugs in the software. Before you know it, consumers will pick up on these bugs, which usually leads to frustration and undermines the brand’s reputation.
That’s why machine learning, which teaches systems to learn and apply that knowledge in the future, makes software testers come up with more accurate results than traditional testing ever could. Not to mention that the probability of error is not the only thing that gets reduced. The time needed to perform a software test and find a possible bug is also shortened, while the amount of data that needs to be handled can still increase without any strain on the testing team.

Test creation:
In Traditional Testing process human creates the test scripts on the assumption and understanding of the application but with help of Data science System can gather the real-time data from different sources identify the hidden pattern and provide new rare critical scenarios also which could help in improving the quality of Product.

With the help of data generated from Data Science Predictive analysis AI can build the test scripts Based on this AI can start creating test cases on real user data. It is smart enough to identify commonly used actions such as logging in/out of the application and cluster them into reusable components. Then it injects these newly created reusable components into our tests as well. Now, all of a sudden, we already have actual tests written by the AI based on real data, along with reusable components that can be used within other tests as well.

Test execution:
Some of the biggest obstacles keeping companies from moving forward with automation is the amount of time and effort it takes to write and execute tests with the chosen tool or framework and the availability of skilled resources to do this task. There are some AI tools that overcomes this issue. Test that use to take multiple weeks can be done now within few hours of time. This is achieved by creating reusable components, run test quickly integrate CI/CD with different grids.
As the test time is reduced the productivity of the test automation is increased and also non-technical person can perform the automation with help of AI.

Maintenance:
One of the most common problems with test automation is maintenance. For example, say we have 100 automated tests running on a daily basis to ensure the main functionalities of the application are still stable. What if the next day we come back to work and find that half of the tests have failed? We would need to spend considerable amounts of time to troubleshoot the failures and investigate what actually happened. This involves figuring out ways to fix the failures and implement the fixes. Then, we re-run the automated tests to ensure everything passes.AI can help here:

Self-maintenance:

Resultant tests are modelled and thus maintained by the combination of an exhaustive and autonomous set of data points, such as the size of element, location on a page, previously known size and location, visual configuration, XPath, CSS selector, and parent/child elements.

Self-healing tests:

Root cause analysis highlights all potential causes for test failure and provides a path for one-click updates.

Data modelling:

Selector maintenance should be eliminated by having elements identified by hundreds of data points that are rated and ranked instead of a single selector.

Conclusion
Machine learning gives testers the opportunity to better understand their customers’ needs and react faster than ever to their changing expectations. In addition, testers now also need to analyse more and more data and they are given less and less time to do that, while their margin of error decreases constantly. Data Science and AI offer a way to address these challenges. This approach is set to fill the gaps of traditional testing methods and make the entire process more efficient and relevant to the users’ needs.

Testing Challenges in Card Management System Migration!!!

Card Management system migrations and conversions are lined up globally across credit card issuers. Credit card domain is witnessing a sharp rise in platform migrations primarily due to:

  • Regulatory challenges – Guidelines from national central banks regarding consumer data placement and security
  • Financial institution mergers – shifting economies resulting in merger & acquisitions in BFSI sector
  • Institutions opting for processor platform instead of in-house processing
  • Keeping pace with changing technologies – Migration to more robust platforms
  • Product upgrades – Platform upgrades with latest version

All migration initiatives have common factor for success – Testing. A comprehensive testing ensures successful portfolio switch over with No to Minimal impact on Bank and customer’s business as usual activities.

This blog will focus on key migration testing challenges and possible control steps.

Migration testing projects typically have 4 phases:

  • Scope
  • Planning
  • Execution
  • Closure

Let’s list down associated challenges at each stage:

Scope:

Challenges:

  • A medium to large scale migration program may have an all-inclusive timeline ranging from 6-12 months. It is practically impossible to freeze process changes on source platform for such duration due to swift changing customer behavior and industry demands. These changes result in scope extension and also impact the next phases – Planning/Execution.
  • Another scope challenge is inadequate documentation. Over the number of years with old platform, a considerable change gets seed in system leaving behind little to No documentation due to repository miss management or resource movement. This creates a scope GAP in certain areas which only surfaces during test execution phase

Control steps:

  • A Business requirement traceability matrix must be maintained with a track of changes during migration
  • Testing teams should be agile enough to include such changes in planning and execution cycle
  • BAU/Business operation teams must be consulted for a broad scope review. This will help to identify any missing process. This approach can act as a supplement of missing documentation.

Planning:

Challenges

  • Planning timeline: Most clients plans to have a larger execution window rather than spending time on proper test planning. This is done in order to reduce overall migration timeline. Shorter timeline limits the comprehensive test coverage and at many instances, plans are revised during execution to accommodate findings.
  • Final parameter set up on migrated platform is generally not available during planning phase and a base parameter oriented planning is conducted. This result in frequent revisit of test plans as soon as final parameters starts flowing in.
  • Batch processing, most of the card processing activities like Due calculation/Payments/Collections etc. takes place in batch mode. This puts an extra load on test planning as different functions will have different batch requirements such as Daily batch/Month End batch.

Control steps:

  • Discussions with product teams can help to arrive at a near right parameter set up on migration platform. This will reduce the planning update during execution phase
  • Always consider batch requirements during planning phase. Cases with specific batch requirements must be clubbed.
  • Testing consultants with prior migration expertise and re-usable test repositories can significantly reduce test planning time

Execution:

Even in ideal scenario, it is not feasible to clear all challenges in scope and planning phase, as a result certain challenges will always flow into execution phase.

Challenges

  • New system readiness – Test environment preparation and readiness sometimes eats up the execution timelines due to interface readiness, correct parameter set up and batch performance
  • Change in solution/business process – such changes impacts the test plans followed by execution plan updates
  • Blocker defects – This area is extremely crucial for execution. Identification and closure of blocker defects has an important role to play in smooth execution and over all test timeline

Control steps:

  • Execution teams must closely monitor system and interface readiness and adjust execution plan as per system availability
  • Change is solutions should be scoped in a separate execution plan to minimize impact on existing execution plan
  • Execution team must set a function wise execution priority and monitor the business requirement traceability matrix
  • All major functions and business process related base parameter set up should be verified at execution start time to identify blocker defects at beginning of execution phase

Closure:

Challenges

  • At this phase, test activities are at its last phase and a sum up of execution results is in process. This phase identities the system readiness to proceed with production implementation. This phase is not encountered with multiple challenges however; one common challenge is Go/No-Go decision. Test team plays a crucial role to assess project state and recommends on the Go/No-Go decision based on execution test results.

Control steps:

  • All open defects must be carefully observed and should be assessed as per severity
  • A walkthrough of all open items should be arranged with working group for final assessment
  • Test teams must ensure that all stakeholders understand the open defects and associated risk
  • A formal agreement should be made for Go/No-Go recommendation

In the end, these challenges may vary on case to case basis. Above expected challenges and its awareness can be helpful to identify associated risks and its mitigation plan.

 

Lets be Agile in Testing!!!

Over the years the demand is growing for the team be operating in Agile fashion and with Agile Testing methods almost at every project location. Project manager wants model or prototype built quickly and testing to be performed in an iterative manner. Obviously, there are advantages of this iterative short cycle testing & the benefits are enormous.

Agile testing method focuses on an incremental, short term deliveries that allows frequent review responses & continuous feedback mechanism. Last week we have seen how Agile mindset helps in project execution, various roles and terms associated with it. In this blog we are going to see how a testing team performs testing using agile methodology. The quintessential difference is that the testing is not done during only one phase, but testing is done iteratively with every release of the software or the product. This way of testing early and repetitively ensures minimal damage and risk towards the end of project Everyone would have seen the classic curve of time vs cost when defect is identified & cost of fixing it. In Agile approach this is exactly the area that testers can best tackle for you. For them to achieve this project manager and project teams must have an agile mindset.

Let’s see how exactly an agile tester can contribute and participate effectively in an agile project environment.

  • Attend Sprint Planning Session.

As a member of Agile Team, quality leads and tester should always attend Sprint Planning sessions. This helps to ensure that they are in sync with the development team from the start and allow QL to identify and highlight possible problem arrears and risk as early as possible. During the Sprint Planning, developers estimate the effort it will take them to write code, similar way testers should estimate effort required for testing code during planning session. So, resulting in a win-win situation for developers and testers both.

  • Attend Daily Stand-up meeting.

QA Tester should attend daily stand up meeting with the developers, Scrum Master and the Product Owner. This promotes a strong collaboration amongst the team members and a sense of one-team prevails. Based on the updates received from the Developers, Business Analysis and Testers, product owner stays up to date with how the sprint is going on and teams are able to plan their workload efficiently. In case, if the tester has a blocker, they can bring it up in the daily stand-up meeting and update every team member on known issue which in turn allow developers to speed up testing progress and plan their workload.

  • Don’t wait to report bugs, report it immediately & frequently throughout the Sprints:

This is very important, in order to deliver high quality of software in a short period of time your team needs to work efficiently. Any issue that is found during sprint must be reported immediately in the same Sprint. Clubbing all the incidents and reporting only at the end does not help in agile setup. If this happens then closing the sprint becomes difficult and makes it ineffective, given that it is a time boxed activity.  By continuous integration of testing and development, it allows both the teams to work together and resolve their issue faster and this help in leading to higher quality of results.

  • Be an active participant of Sprint Demo

At the end of each Sprint, tester and developer teams demonstrate the product features that are developed to the product owner and other stakeholders. Any feedback has been received related to look and feels & other features that is incorporated on immediate effects. This help to deliver the product with high quality customer satisfaction and meet the business requirement.

  • Document the test cases

Although in agile project where documentation is minimum, due to frequent requirement changes, testing team cannot afford to skip necessary documentation. Documentation is very important for QA. Execution for test cases & pass logs is the proof of successful testing and is enquired to be documented at various levels. This becomes useful for Bank internal or external audit requirements. This also covers the scenarios where a person is replaced by another one.

  • Attend Sprint retrospectives

At the end of each sprint, team needs to discuss and provide review comments about what went well & what did not work during the sprint. This helps to improve in future sprint planning and execution. Sprint retrospective is the opportunity to define weaknesses and determine solutions for them. QA need to be involve in these discussions for any concerns that needs to be addressed before the next Sprint can start.

There are certain cautionary warnings that must be taken care during the agile testing:

  • Avoid delays in delivering a story point. This will have cascading effects on QA, testing and eventually in achieving the sprint goal.
  • The scope and vision must be clear for a sprint. Any changes identified during the sprint should be parked & then incorporated in the next sprint. There is always a temptation to include everything in the on-going sprint. This approach can prove counterproductive and losing the sprint altogether. Hence must be avoided.
  • Try to avoid falling in a trap of using Agile as an excuse for missing documents. Agile does not mean no documentation; bust it means minimal documentation that is essential. Ever increasing scope and improvement feature, just because it is agile project one should not keep adding features, wish lists and ideas to the existing project. There must a definite start and an end. Otherwise the project can become an eternal agile project.

The contribution of agile testers in various ways listed above, results in multiple benefits such as:

  1. Improvement in successive Sprints: By producing frequent builds and conducting testing and reviews in an iterative model, there is an opportunity to fix at every instance. Thus, early detection and fixes result in improvement of product over successive Sprints.
  2. Early Prototype: Everyone gets to see how the end product will look like, any changes can be quickly incorporated.
  3. One Team Approach: Due to daily scrums and frequent interactions the team develops a truly one-team bond that delivers better and faster.
  4. Working software: Getting a working software is the Key. Planning and building for months resulting in an inadequate software is the nightmare for any project team. This risk is very efficiently handle due to agile testing approach

Knowing all these warnings signs & benefits, project teams reap the benefit of successful product as end result. The It is flexible, quick to see results, quick to incorporate and implement changes. Minimal essential documentation and high-quality software / product output. If we compare these things with a traditional approach one can easily see the immense benefits. So, let’s get going and Be Agile!

Be DevOpstastic !!!!!

DevOps!! The new trend coming in the Industry….

Let’s see the whole world of IT Industry with the lens of DevOps. DevOps does not go with only one thing, it has got the combination of Continuous Integration, Delivery, Continuous Deployment and more. DevOps can give the beautiful experience to the Industries handling large customer volume, bringing more enhancements on Automation services and Product Life cycle. As every business needs to stay agile, wants to improve their speed and innovate faster, DevOps methodology supports quality assurance and risk management factors.

DevOps equals Continuous Delivery. DevOps is a (combination of development and operations) software development method in which the operations and the development engineers participate together in the entire service lifecycle, from design through the development process to production support helping an organization rapidly produce software products and services. DevOps practices arena started nearly around 2007.DevOps innovate faster.

Dev and Ops are the two different terminologies which shows that the developers work with Ops to understand the impact of code changes also developer works more closely with production equivalent systems.

DevOps an evolution in the Software Delivery and what does it address?

Several times we have come across conflict conversations like “It’s not the server, it’s your code”, “It works on my machine” etc. What can be the major reason behind this conflict?? There might be the lack of synchronization between the development and the operations or infrastructure team?? Well, the answer to this question is “Yes”. The above conflict can be reduced by adopting DevOps. The development team is responsible for the creating and modifying the code on the other hand the operations team is responsible for creating stability and enhancing services on the developed code. With today’s constant pressures to provide new and innovative services—IT organizations must encourage and institute a culture of continuous learning and improvement. DevOps, a relatively new approach that draws on agile IT development but goes far beyond, can help IT achieve these goals.

DevOps brings out the collaborative mindset between Devs and Ops covering all the aspects of functional and non-functional requirements of an Industry. The complex banking applications takes months or up to a year to develop and launch. However, with the adoption of DevOps methodology and the tools the entire process can be completed in the matter of weeks after rigorous testing.

Before adopting the DevOps methodology, the development and the operations teams used to work individually for the launch of the application. After the introduction of the DevOps the teams collaborated with the single shared objective enabling Agile development and continuous delivery. The API’s introduction made the development process and helped in enhancing the existing services. Performing the non-functional testing approaches from the cloud became a part of it. Thus, DevOps has got great advancements within itself.

Agile and DevOps – How is it connected?

How Agile and DevOps works together is now a question in our mind. Basically, the answer could be “DevOps is an extension of Agile Methodology” Or “Agile Methodology works for DevOps software development processes and product/application delivery enhancements”.

The end to end Automation and the Collaboration between the Dev and Ops teams are the two key ingredients for making Agile and DevOps successful. The question also arises that what can be the use of Testing in DevOps? There are various testing tools (Jenkins a CI tool) and the activities which we need to sink along with the development process in order to help or to speed up the application delivery in a transparent manner. When the developer deploys the build to the QA and the QA has to run the test automatically providing the test results to give an indication where the build can be deployed to the production. As always said finding a bug in an earlier stage is less expensive. Hence, the testing is triggered automatically when the build is available. The diagram below shows how the Agile and DevOps works together hand in hand.

Agile deals with the processes like scrums, sprints etc whereas DevOps deals with the technical practices like CI etc.

How DevOps solves the problems that the Banking Industry is facing????

We all know that the many people are doing the online banking like never before. According to 2016 MX customer survey, clients find it more important to have an easy digital banking experience (67%) rather than a friendly teller or staff (33%) when choosing where to open an account and contact their banks mainly through mobile devices. The 3 main priorities that have been listed by the banking IT executives to enhance the customer service experience and banking software’s: –

  • Redesign or enhance the digital experience
  • Find ways to reduce operating costs
  • Enhance data analytics capabilities to enhance customer needs

In the highly competitive banking industry, we always look to leverage the power of Software. As IT is becoming the heart of the Banks, with DevOps we can have it all (The cost, The Time, The Quality and The Delight). The large automated banking applications and the team have noticed an integration issues with the older codes. The Continuous Integration (Each new application build is automatically deployed and regression tests are triggered against the build) and the automation implementation ideas can bring out the best in the DevOps delivery chain.

Well with all the advancements DevOps is becoming the predominant industry solution. In summary, DevOps is evolving rapidly and each of these pieces provide a slightly different outlook on 2017

8 KEY QUALITIES OF AN EXCELLENT TESTER

“In God we trust, the rest we test” is a very popular line in testing world. Testing team or testers are a very important cog in the wheel as far as product development life cycle is concerned.

A tester essentially conducts prescribed tests on programs and applications for quality, design integrity, and functionality, before they are implemented. A good software tester applies punctilious testing methods and considerable end-user simulations to expose program bugs so that they can be eliminated by the software programmers.

What makes for an excellent tester? Along with possessing a very refined skill set that allows them to do their job well and add value to a development project, a software tester needs to have a few qualities that will make him excel in his field.

Although this is not a complete list, I believe following are traits that an excellent tester should have:

  1. The ability to analyse data and be curious:

Creating scripts, and executing them numerous times is not a big deal. A good tester does not limit himself to just testing, he analyses and understands the data collected from testing to study the specific behaviour of the application or the product. He goes in depth and fishes out the root cause of bugs and thoroughly analyses the test environment, test data, interruptions etc. He should find out everything about the product under test.

  1. The knack of good reporting:

A good tester meticulously works for hours together to execute a number of test cases, mark them in management tool, and project a status. An excellent tester along with that, knows exactly how to report it to be understood best. He reports the status to the clients in few crisp sentences as to what bug numbers he found and what will be the next action and not just how many tests were done during the day.

  1. The will to be a constant and quick learner:

A primary difference between a good software tester and an excellent one is the former is satisfied after gaining expertise while the latter leaves no stone unturned in keeping up with the latest in technology and automation tools. An excellent tester learns to create new ideas, and then form an experience that comes from constant thriving. The ability to grasp the essence and do it quickly is absolutely key.

  1. The ability to give attention to detail:

Often a software tester only has his eyes set for the glaring issues but fails to look past the obvious and find out small things that can have a trickle-down effect on the whole project. An excellent tester therefore is the one who pays great attention to details and picks out the little issues that affect the big picture of the project.

  1. The flair to communicate well:

A software tester is often in very close contact with the developers, business analysts, and the major stakeholders in a project. Effective (verbal and written) communication skills therefore are necessary. An excellent tester has great communication skills and therefore he has the ability to understand requirements, describe test criteria and is able to explain how to recreate issues. Apart from good communication skills, a tester also need to have the courage to vocalize concerns and ask doubts.

  1. Ability to think outside the box:

Any tester can get involved in functional tests and use cases but it takes an excellent tester to push the envelope and strive hard to think outside the box. Users always don’t end up handling the software just as intended. Thinking about situations that aren’t ideal, analysing environmental conditions, and doing case based test on applications are signs of a great tester.

  1. Highly adaptable to varying situations:

When a project is moving rapidly, requirements sometimes can change in a jiffy, features can shift focus and deadlines can move forward or backward. An excellent tester therefore has to be highly adaptable to different situations.

  1. Ability to code (programming skills)

An excellent tester will have the capability to at least read and code as to understand the intricacies of product under test. Also the reasonable amount of programming skills will help during automation testing as tester can create tools and snippets of codes needed to automate the manual testing.

The movement of the software industry has been rapid in a short span of time. People’s attention span has become shorter, there is a steep rise in the need for faster deployments and it has become tough to keep tabs on the rapidly changing technology. The industry’s need for quality testers is fast increasing and in this tough game, only the best will survive to do work that is truly worthwhile.

Any important or additional traits that need to be part of this list?

DO TESTERS NEED PROGRAMMING SKILLS?

I have been leading Verinite Labs for 6 months now where the mandate has been to develop our own IP by building technology solutions for some of the problems faced by the financial institutions. Being from services background, these 6 months have given me an interesting perspective about lot of things and one of them is about an off-running argument of “Whether today’s testers need programming skills?”.

In a product team, a tester plays a unique and an important role. He is an advocate for the end user while collaborating with his team in maintaining the right agility in its operations and the product development effort.

To be able to understand the product under development more technically, to automate tests, develop test frameworks, and customize tools to meet the team’s testing needs, programming skills come handy.

Testers come under 7 categories:

  • Administrative
  • Technical
  • Analytical
  • Social
  • Empathetic
  • User expert
  • Developer

Every tester type has its own value, plus a tester does not have to confine himself to a particular category.

Usually, most testers come with an engineering background, and the least coding skills. They lose touch though if they don’t get sufficient opportunities to code.

Testers can, if they wish, ensure they don’t let their coding skills rust. For that they need to do periodic static code reviews, work on test automation, review existing test automation code etc.

Some argue that since all testers don’t do test automation, all testers need not have programming skills. Because testers who do exploratory testing, bring along a different yet valuable set of skills to the team. They have good analytical skills, investigative skills, and a deep understanding of risk, and other hidden issues along with some technical knowhow like databases, networks, system administration etc. Some of the very best testers don’t know even a bit of programming.

Some others argue that coding and testing skills should not be mutually exclusive.

They say that if a tester has programming skills it will benefit him and the organisation immensely.

Here are a few benefits:

  • Ability to code, makes the tester comfortable in owning and preparing test environments. He can pick out the problems, ask for help, and resolve matters more quickly. He is also more confident of dealing with web servers, DBs, Message queues etc., and can write scripts to maintain, monitor, and prepare test environments.
  • If the tester is comfortable with coding, it is easier to drive testability. Even a fairly good understanding, of say what might make underlying code more testable, can help a tester discuss test only end points, hooks in the system, and test only configurations with the developers. He can also appreciate the limitations to implementing them if any.
  • Programming knowledge helps a tester see the boundaries and layers of underlying systems more clearly. This helps him avoid duplication of test efforts as well as gives him an opportunity to focus on things which are important to test from the system’s point of view.
  • The Ability to code gives the tester a better understanding of the release process. He can appreciate the different branching strategies and can see the importance of continuous integration. It can also aid discussions of merits and risks of continuous deployments. It becomes easier for him to own continuous integration and automated testing.
  • A tester with programming know how can better understand the complexities of development. They find it easier to spot risks, find problems, and speak the language of developers. They are more in the position of contributing meaningfully in technical discussions.

So there are many reasons why skilled testers should build their programming skills so that they are able to read, write and understand code.

In the end, with areas such as artificial intelligence, big data and interactive computing, manifesting themselves in several domains, technology is growing at a very rapid pace. So in today’s world, the value and the versatility a tester brings with his programming skills will go a long way in differentiating him and his testing team. So it makes more sense to start understanding the value of a tester with programming skills rather than making it mandatory for testers to now programming.

CHALLENGES IN TESTING THE BANKING APPLICATION

Challenges in Testing the Banking Application

In a continuous endeavor to leverage technology, improve Cost-to-Income ratio, and enhance the services offered to customers, banks keep upgrading their systems and introduce new customer touch points. Innovative products and solutions are launched to make banking even more safe, secure and convenient for customers. With increase in complexity of these systems, testing of the banking applications becomes more and more challenging.

Typically a banking application has to undergo following testing to ensure that it meets the goals:

  1. Functional Testing
  2. Security/ Penetration testing
  3. Performance Testing
  4. Compatibility / Usability testing (for Applications with Customer facing Interface)

Let’s discuss some common challenges faced while conducting each of these testing.

Functional Testing:

  • To meet the ever changing business need of banking industry, the applications are made highly configurable. Introduction of new parameters increases the number of functional paths that should be verified during testing by manifolds. All these possible scenarios along with the boundary conditions should be rigorously tested. This increases complexity in Test design phase. Covering all the relevant and critical scenarios within the stipulated time of a project can be a daunting task
  • With the increase in customer touch points like internet banking, mobile apps, self service kiosks to list a few, testing should make certain that specific scenarios for all access channels are thoroughly covered. Simulation and capturing of test results for all these channels is an eternal challenge
  • Compliance and regulatory requirements vary from region to region. Any change in these requires regression testing of the system. Understanding the changes and translating these changes into system impact and test scenarios can be complicated at times
  • Lastly, importance of proper data can’t be undermined for a complete and effective functional testing. However many of the times it is difficult to prepare the required input data to simulate all scenarios

Security / Penetration Testing:

  • Owing to the sensitivity of data, banking applications are most vulnerable to hacking, spam and other fraudulent activities. As more number of end points are added to an application, more avenues are presented to the hackers through which an application can be attacked. To ensure data security, all these channels and the types of accesses needs to be tested thoroughly
  • Different countries have different bank secrecy laws for customer data protection. Moreover, international data security norms are updated regularly to keep the standards abreast with new security threats. This makes the testing even more essential and tough

Performance testing:

  • Most of the performance testing scenarios can’t be simulated in test environment, therefore a near simulation or simulation through some external tool is used for testing. This limits the effectiveness of testing as the actual issues can’t be traced which might arise in production later
  • Normally the test environment does not have the same processing capacity as the production environment. So most of the time some issues are attributed to the limited capacity of the test environment. For a tester to prove that the errors are due to some flaws in application and not due to the limitation in test environment is very difficult
  • Moreover, creating and maintaining the extensive range and volume of data similar to production is a tedious task

Compatibility and Usability Testing:

  • The ease of usage of any customer facing application decides its fate. The same applies to banking applications as well. The application should be user friendly and should be accessible to bank’s targeted socio-economic segments. In some cases the success of an application can help the bank to acquire, retain and grow their customer base. This aspect should be envisaged and covered in testing
  • A customer can access a banking application using different platforms like different browsers, devices or networks. In order to provide an optimal user experience, bank needs to ensure that the look and feel of applications on any of the access medium remains the same. Hence testing should cover compatibility across all platforms while adhering to their security standards. Also, scalability of the design to accommodate new delivery channels should be considered. This makes the scope determination and execution of compatibility testing for any customer facing banking application a massive task

After considering all these challenges, there is no doubt that software testing for banking application is an arduous task, where so many things can go wrong.

But, as per the famous quote from Mosher’s law of software engineering:

Don’t worry if it doesn’t work right. If everything did, you would be out of a job.

HOW TO ENSURE QUALITY OF TEST DATA!

Test data helps us examine a software application and determine its functionality. The data used to test the features of a newly developed software program is simply, thus, called test data. It helps identify bugs provided it is properly designed. If the test data itself is faulty or of low quality, it will not do a good job identifying bugs and testing features. Needless to say, it will directly affect the quality of the software.

Generation of Test Data

Since the test data determines the quality of software testing and the development of the software application, you need to keep a close eye on its generation. Generation depends on the type of testing environment and software to be tested. It can be done in different ways:

  1. Manually
  2. Using automated test data generation tools
  3. Mass copy of test data taken from client systems
  4. Mass copy of test data from production to testing environment

Please keep in mind that test generation is quite complicated and requires time. In fact, in most cases, test generation takes a lot more time than the testing itself. However, it is mandatory that you pay close attention to this step because it will affect the quality of test data, thereby affecting the software quality.

Ensuring Quality of Test Data:

White Box Testing

For this type of testing, the test data can be extracted from the code directly. The quality can be ensured considering the following:

  1. All program source code branches should be tested at least once.
  2. All paths must be tested at least once.

Black Box Testing

Since the tester cannot see the code in this case, certain criteria should be met in order to confirm quality of the test data.

  1. No data: Do not submit any data (blank data) and see whether the system responds with error messages.
  2. Valid data: Submit valid data and see if it is being saved. Also check if the system is functioning as expected.
  3. Invalid data: Submit invalid data to ensure proper behavior for alphanumeric inputs and negative values.
  4. Illegal data format: The system should generate error messages and refuse to accept the data format.
  5. Boundary condition data: This will help identify response to lower and upper boundary conditions.
  6. Performance, load, and stress testing data: This data set will be voluminous and will help you determine the efficiency of the test data.

Security Testing

Security testing revolves around protection against malicious content. The test data designed to test security software must satisfy the following:

  1. Confidentiality
  2. Authenticity
  3. Integrity
  4. Authorization

Likewise, parameters to ensure the quality of test data vary with the software application. If the applications are complicated, going with automated test generation tools is the safest way to guarantee quality. As aforementioned, it is the quality of the test data that will influence the quality of the software. That is why putting it under the microscope is highly recommended.

DO YOU REALLY NEED TEST AUTOMATION?

Do you really need Test Automation?

Last week we were having a call with one of our Clients about some additional Quality Assurance services that our company has to offer. Test automation was also a part of it. After a fifteen – twenty minutes of detailed discussion on the additional testing services, they asked us a simple question “Do we really need Test Automation?” The question was a straight one and they were expecting a “Yes” or “No” as an answer but for us the answer was not that simple.

automation1As per the data available in public domain, around 63% of all Test automation projects have failed to achieve expected goals and around 50% of procured test automation tools have never been re-used after completion of a project. With these facts in mind, one can easily consider Test automation to be a rather risky proposition. But before we draw any conclusion, we need to consider how the successful projects (remaining 37%) made it work in their favour.

For any successful implementation of Test automation, it is critical that we evaluate and confirm whether test automation is actually required. There are so many myths built around test automation which most organizations believe it’s a panacea to all their quality related ailments. So we need to look beyond those myths and conduct a proper evaluation to determine its usefulness

An example: Most of the time the very objective of starting a test automation activity is to save time, cost and efforts in a project. But on the contrary Test automation involves higher initial investment to procure tools and conduct training, needs specialized resources (hence effort) and additional time to prepare automation scripts.  The benefit of test automation can only be realized over a period of time depending on the reusability of scripts and frequency of execution.

Therefore performing a ROI analysis on each automation initiative is an objective way to conclude the type and the extent of automation that is required / beneficial for a project.

automation2A simple analysis can be done by executing these following steps to determine the need of automation for a project.  (There are some cool tools available in the market to conduct this ROI analysis, but my suggestion is use an excel sheet with your own parameters. It’s much simpler, quicker and cost-effective way to come to a conclusion)

  • Step1: Calculate Potential Effort and Cost savings through Automation

Following points need to be considered while coming up with total potential effort and cost saving figures

  • All activities in a Testing life cycle should be considered. Remember, test automation is not limited to test execution. It can also be used to reduce effort for activities like test environment setup and test data creation
  • Break down all these activities to combination of independent task groups
  • Consider the frequency and number of repetitions for each task group as per Test plan and Test scripts
  • Assign a reusability index (i.e. probability of reusing the same task in future projects) for each task group
  • Step2: Calculate Total Cost of Automation

Following points need to be considered while calculation total cost

  • Technology requirement for automation of tasks
  • Cost of Tools that supports automation for these technology
  • Skill requirement for implementing test automation
  • Overall effort for building automation scripts for the required task groups
  • Step 3: Optimize the scope of Automation
  • Most of the time organizations end up trying to automate each and every step of Test cycle
  • Now if we go by this approach, more often than not we’ll end up having a negative ROI result. Because automation only make sense when it can be reused over a long period of time.
  • Therefore we need to prepare multiple automation models and subject them to the ROI calculation process to come up with an optimized automation model which can be implemented for the project.
  • Automation models are created by varying the scope of automation from everything to nothing. Different task groups with less frequency, less repetition and less reusability index can be removed strategically from the scope of automation to come up with multiple models.

Based on the above process an optimized Automation model can be derived for the project, which can add value and help to achieve the goal of automation for the client successfully

Now that we’ve decided an optimized automation model for the Project, the next step is to implement the same. Next time we’ll discuss the approach for successful implementation of Test automation in a project.

In the meanwhile please feel free to share your queries and views on the topic through comments in this blog.