No matter how many efforts you apply to ensure Continuous Testing runs smoothly in your company, you may be a little disappointed by its results. What did you do wrong? Read our article to find what points you may have missed when you started applying Continuous Testing practices to your processes.

Tricentis states that in 2019 Continuous Testing will become mainstream. Has it? Oops, nope. Yet again, we’re discussing the obstacles that companies willing to apply the best testing practices will find on their way to success.

Risk-based Strategy of Continuous Testing Vs Pass/Fail Results

The charming thing about test automation is that it, like a crystal ball, does not mystify its answers, but gives clear pass or fail results. As it goes, this is also its weakness. Due to its single-dimensional nature (only pass or only fail), the problem parts of the application – its major risk-prone components - cannot be scrutinized as through TA we perform sort of a high-level assessment of the wellbeing of the system.

As Wayne Ariola specifies in his study, risk-baseness is the crucial differentiator in terms of achieving CT objectives. According to his viewpoint, such change in strategies stems from the fact that “Business don’t want… perfect software. They want to deliver new, business-differentiating software as soon as possible”. Customers need the quickest feedback on whether top-notch technology will act according to the expectations or fail in production. Good old test automation often cannot meet such demands, as, for instance, 5 testers may be working on the project and “doing just fine”, but in reality they test the wrong things or the wrong set and TA simply doesn’t work.

Why does it happen? Generally, most tests are written to give low-level details on whether user stories properly reflect the requirements. Such attitude toward testing is too broad, as the results won’t tell you if the app is too risky to release. "Would you automatically stop a release from taking place based on test results? If not, your tests aren't properly aligned with business risks”.

How to address this issue? Mr. Ariola recommends that testers 1) understand the risks related to the whole application portfolio; 2) map risks to application components and requirements; 3) apply a test suite that ensures maximum possible risk coverage given the least amount of test cases is run; 4) report status that indicates risk exposure from any possible perspective.

Shift-Left Testing Vs Shift-Right Testing

It is very convenient to apply well time-tested shift-right testing along with traditional test automation. However, today, as a new testing era approaches, it is most impractical. How to modify your shift-right testing procedures then? Combine them with brand new shift-left ones!

Study the comparison of the two testing methodologies below to see why you need both of them.

Why choose shift-left testing?

Firstly, as this testing methodology implies an earlier start of the testing procedures, the bugs critical to a risk-free release of the product will be identified earlier. Early bugs detection equals lower costs to fix them. How much exactly will you save? The Systems Sciences Institute at IBM states that fixing the bug detected after product release will cost about four to five times higher as compared to that found during the design phase. If found in the maintenance phase, to fix the bug will incur additional costs of up to 100 times higher.

Furthermore, shift-left testing narrows the gap between development and testing as “it allows artifacts to be properly validated through clearer and better communication” (X-Ray).

On the flipside, however, shift-left testing leaves UI part somewhat unattended and cannot ensure proper end-to-end validation.

What’s so great about shift-right testing?

First and foremost, it enables getting direct feedback from users. Second, shift-right testing guarantees a more profound automation testing which saves you a lot of troubles as now you can run real performance and security tests and automate UI tests. Additionally, it allows performing solid end-to-end system validation. What’s wrong then? Ah, it’s all about the money, as usual. If a bug is uncovered at the final stage of the project, it is excessively costly to fix.

As you can see from the description above, no methodology is impeccable. Leveraging the benefits of the both will ensure high quality of the system and, consequently, a long and trouble-free life of the product.

Hastily Written Requirements Vs Meaningful Requirements

According to The Definitive Guide to Continuous Testing, 64% of defects occur in the requirements phase. Why so? Since when the load of responsibility shifted from testers and developers to those who failed to elaborate descent product requirements? Frankly, since always. High-level requirements result in poorly developed applications and only loosely tested code strings.

Requirements written in a high-manner fashion are a pretty common notion. However, such shallow explanation will suffice for a developer or a tester with domain subject matter expertise that will allow them to come to the appropriate interpretation of the requirements and fill in the gaps with their own knowledge. Nevertheless, puzzle is not always assembled perfectly.

To avoid such outcomes, elaborate your system requirements in sufficient detail, so that the technicians won’t interpret them differently. Beware also of extremely specified requirements. Developers and testers will unlikely digest the volumes of specifications with every small detail of the to-be system.

However, a change in requirement writing patterns will work better if not applied alone. The authors of “A systematic approach to risk-based testing using risk-annotated requirements models” urge the developing community to apply risk-annotated requirements models before starting code writing. A study by Mr. Ariola was mainly about uniting developers and testers in order to combine their mutual efforts in fighting risky bugs. This time Marc-Florian Wendland, Marco Kranz and Ina Schieferdecker specify how to guide the developers in their work so that they initially create a risk-resistant system.

System requirements specifications are still mostly textual today, which is the root of many problems due to “informal or imprecise textual specifications or the lack of human beings to grasp complex and comprehensive textual specifications”. To make development more risk-oriented from the start, the authors propose to apply Behavioral Engineering methodology. It allows developers to derive formalized requirements models out of textual requirements specifications, which are often informal and contradictory. Under this methodology, risk-oriented and non-ambiguous requirements are obtained through requirements formalization and integration activities. The two activities allow obtaining high-quality requirements and create a complete reliable overview of the compositional and behavioral designation of the system. Now developers are empowered to do their job safely with a clear view of what’s expected from them in terms of risk-exposure.

To sum it up…

Along with the above mentioned risk-based strategy, shift-left testing and meaningful requirements, Continuous Testing is impossible without proactive talents ready to wear many hats. The employees that invest their best efforts into unveiling true risks instead of generating high-level pass/fail results. Top-notch solutions to empower the developers to do the testing part are also of great importance. However, the initial costs may scare a good deal of companies away (more about the Impediments to Continuous Testing here).

What to do if once you set up your mind on applying CT philosophy to your existing procedures but end up failing? In addition to our insights, see the matrix below to find more about where you are now on your way to 100% CT-adoption and what steps you should take to ensure your journey is sustainable and consistent.

Source: Continuous Testing: Transforming Testing for Agile and DevOps

Moreover, you can study the techniques recommended by The Definitive Guide to Continuous Testing, which in our opinion also gives very handy insights on how to make a proper transition to the CT-processes.

Need more information about our Test Automation practices? Contact our Support team or visit our website. Qulix Systems is renown for its QA and Test Automation services and pays immense attention to the professional advancement of its testing engineers.