One of the popular questions from our clients in early adoption stages is:
“Before, we had a clear focal element of each scenario, so we knew exactly what would fail the test. Now, with all the inputs mixed together, how can we identify the cause of a defect?”
The purpose of this document is to demonstrate the slightly different analysis approach using Hexawise tests. We will use a simplistic customer preferences application with 5 “Checked”/ ”Not Checked” indicators, looking like this:
The traditional, manually selected tests would likely focus on isolated scenarios:
So, if test #3 fails, there is likely something wrong with Indicator 3.
If we use Hexawise to generate the default set of 2-way tests for this application, we will get the output below:
Now, if we look at test #4, it is not just Indicator 3 that is checked, but also Indicator 1. So, if the execution fails, how do we pinpoint the reason?
The key is to look at the whole suite and identify the dependency between the consistent elements & the scenario outcome.
First, let us define what is defect strength. We will call an issue “one-way defect” if it is caused by a single parameter value irrespective of everything else happening in a test case. Two-way defect means that a specific combination of 2 parameter values triggers the error, and so on.
Keeping that in mind, let us go back to the original manual test #3:
If this is truly a 1-way defect of “Indicator 3 = Checked”, then the rest of the values in the row will not matter. The same defect would be caught by the following highlighted Hexawise tests:
The debug process in this case would consist of the following steps (assuming there is no error message specifying what went wrong):
- Execute all tests, notice that #1, #4, and #6 were the only ones that failed.
- Analyze what is the consistent element of these tests:
Indicator 1 has both values – not a culprit
Indicator 2 has both values – not a culprit
Indicator 3 has only “Checked” value – culprit
Indicator 4 has both values – not a culprit
Indicator 5 has both values – not a culprit
Therefore, the holistic view of the Hexawise suite should provide you with enough data to pinpoint any n-way issue.
At the same time, the original manual suite did not cover two indicators checked simultaneously, which means there is significant risk of 2-way defects (or higher) slipping into production. Based on the statistical evidence, on average, 84% of defects in production are caused by 1 or 2 system elements acting together in a certain way.
Thus, it is important to use combinatorial methodology to improve the testing coverage and then adjust the analysis process accordingly.
However, there are situations when the isolated testing is absolutely necessary. Hexawise also allows you to perform that but requires more advanced feature usage. Thus, you can implement requirements to force only 1 indicator to be checked at a time in the first 5 scenarios:
Then the rest of the suite will take care of the pairwise interaction coverage.
Alternatively, you can use higher coverage strength, especially if the system has no more than 6 binary parameters. In our demo example, selecting “5-way” from the dropdown automatically provides us with all possible combinations of 5 inputs. You can read more about the interaction coverage strengths here.
If your experience with defect analysis has been different, we would appreciate you reaching out & telling your story so that we could improve our approach.