Product Management Automated testing

Double hop in: when to start automated regression testing for cloud ERP

By Bartosz Szpiech on September, 29 2020
7 minute read

Stay up to date

Back to main Blog
Bartosz Szpiech

Bart is a Microsoft Dynamics fanboy. In 2019, Bart took over 130 international flights to help businesses solve hundreds of issues and show other ways how to assess the problems. Aside of test automation he specializes in Security Role based configuration in AX2012 and D365 providing full project leadership in that area.

"We are free to choose our paths, but we can't choose the consequences that come with them."

- Sean Covey, The 7 Habits Of Highly Effective Teens


For many reasons companies combine ERP implementation with their cloud transformation - starting the cloud ERP implementation or upgrade project. They are beginning two exciting transformations marked with profound change in the scale and the way they test regression. Let’s check 3 scenarios for automated testing integration.


Going cloud   

There’s a life after go-live in the ERP project. 

In fact there always was. ERP from its definition should live, change, adapt to business opportunities, reflect risks, etc. The company’s spine cannot be stiff. It means a lot of customization, updates, cross-platform integrations. It thus means a lot of updates produced, tested and integrated into basic, standard code. Any change in one process has consequences for other processes. Each change should therefore be proved on consistency of the entire processes logics thru regression tests...


cloud environmentERP update management was always a true test to IT-business relations effectiveness. A great adventure for those who know the ropes.

But once you decided to go cloud and choose one of the cloud ERP offered so passionately by global vendors - Microsoft , SAP, Oracle…  you name it... this after-go-live phase will become a completely new experience. 

There are many aspects and perspectives of this change. The clue however might be again a change of updates amount you will have to manage. Yes, your organization contribution will remain an important source of change. 



cloud ERP is a new normal. 

It is Your cloud vendor who gives the main beat now. Your ERP will be updated on a ca. bimonthly release pace  - in case of Microsoft Dynamics 365 - for example. The schedule of releases is set and defined. And yes, it’s pretty intensive. Just imagine testing manually hundreds of processes for regression! 

You still have a choice how to respond to it, of course. You may become for example a Dynamics 365 version-one bimonthly up-to dated ERP company. What you gain? A state-of-art corporate engine supporting effectively and stably your business.


Once you consider and calculate the risks and costs however you may decide to update less frequently - every 3 or even 6 months. In both cases the problem of regression tests remains - there’s only a time lag in second scenario.

You may also figure out to update even less often, irregular or even hesitate to do it. But be sure - your cloud pace-setter (or even pace-maker) will force you to abandon this strategy sooner or later.


Once you - dear test manager, ERP project manager, CIO - face the cloud release schedule new normal, the key issue will be still the question of how to deal with update management. Sooner than later you will have to calculate the


man-day amount dedicated to regression testing. 

It may happen that you decided to update every three months and, besides, your ERP implementation is as standardized and simple as one can imagine or hope to. You need 3-4 man-days for test regression each quarter. 

ilya-pavlov-wbXdGS_D17U-unsplashIn other case you will require from 10 to 20 man-days per update. This is a standard span of testing demand for a typical customer. Depending on how often you update this means from 40 to 80 man-days per year while you update quarterly. Or 120-240 mandays yearly if you decide to be a version-one company. Don’t forget to add your “organic” update stream source - further customization, new functionalities, etc. Sounds gigantic? 

Not, when you 


opt for test automation. 

Automated regression testing tools are a long established solutions for many areas. However that's a quite new phenomena in a cloud ERP world. For example, every Microsoft Dynamics 365 customer, and thus its test manager, ERP project manager, CIO, etc., will soon recognize that dealing with continous-development of application encouraging approach and ca. bimonthly vendor update cycle requires automated testing for regression.


What to choose? Smoothing and streamlining update process for their Dynamics 365 customers, Microsoft delivers its own, free Regression Suit Automation Tool (RSAT). Looking for a better usability, effectiveness, functionality and testing scope one may choose independent tool designed to test regression in Dynamics 365 environment, Executive Automats developed by XPLUS. 

Whatever you decide to choose however, first and most important question is: 


when to start testing?

There are three scenarios for integration of automated regression testing of the cloud ERP implementation or upgrade.

  • First: early adoption of testing tool.
  • Second, as opposite: in post go-live phase.
  • Third, compromised: close to go-live phase adoption.


Let’s  analyze all three scenarios starting from the last one. 

Your cloud ERP implementation project is close to an end. Your processes are almost ready, standards are chosen and approved, the code is freezed and you prepare the user acceptance testing (UAT).

This is a critical moment of the entire ERP implementation. Before you step into UAT phase  it would make sense to run an automated testing tool. The scale and number of the tests will achieve its pick in that phase. In order to mitigate risk of non-acceptance from users, you will have to run hundreds of tests, both functional and performance before. UAT will also reveal lots of bugs and your automated testing tool will be an essential tool to match consistency after you will fix those bugs.   


Automated tests designed in this phase will be useful in your life after go-live adventure. Your ROI from automated regression testing tool will be provided faster. Timing seems just perfect.  

You may delay your testing automation tool adoption  into post go-live phase. It seems reasonable and natural as you are already aware of the forthcoming vendor and internally generated updates cycle.  This is second of possible scenarios. 


It might be an interesting option as test managers or CIOs always decide to go hybrid  and  balance automated and manual regression testing. After go-live you see clearly total portfolio of processes included into your new ERP solution. It’s much easier now to calculate the risk, the cost assessing criticality and importance of processes and forecast the demand for changes, and updates in the future. 


Now it’s the time to design your own, customized future testing strategy. 

Some don’t want to wait that long. 


In fact, they need to start with automated testing as soon as possible. And, depending on process and standard, it is possible. You can choose and adapt automated testing starting with simpler, less important ones. The bunch of tested and proven processes, the safe and checked area of the new ERP environment, will grow systematically. 


will-h-mcmahan-S3JdHNXSfnA-unsplashWhat is crucial, however is to choose early and complex automated regression testing adoption is a know-how. Processual self-consciousness of the organization itself and deep, experience-based knowledge of the implementation integrating company. One should have capability to predict correctly which processes and how may interfere. If you are skilled and savvy veteran you are brave enough to clear and finish things quickly.

Suppose, it’s a question of the scale of your matured and tech-advanced company; you don’t want 3-years ERP implementation projects anymore. That’s why you look with sympathy at automated testing vehicle, appreciating it for usability, flexibility… Once you will find bugs in the process already automated, you find it necessary not to teach robot and build testing scenario from the beginning but “edit” existing script.  



what’s your scenario?

This last clearly is an option for most advanced companies with strong IT - business relations. You probably also choose heavy-weight integration company who can share with you best practices and experience.


Since you are in middle of the rank, your company focuses on business not technology, but never underestimates your efforts to provide best value for  business, you probably should decide to start automated testing for regression around go-live phase. 


Those who need more time are able to calculate and adjust it to business and financial capabilities will probably be more secure starting it’s automated testing adventure after go-live phase, but still on the doorstep to new-cloud-defined-ERP-normal pathway.


Frame 27


Stay up to date