This is an overview and explanation of a set of functional tests created in the Hexawise test design tool.

What are our testing objectives?

Each time someone applies for insurance, thirteen different test conditions are included in the test scenario. Even in this over-simplified example, there are over 300,000 possible scenarios. In this context, we want to test this loan application process relatively thoroughly - with a manageable number of tests.

We know that testing each item in our system once is not sufficient; we know that interactions between the different things in our system (such as a particular credit rating range interacting with a specific type of property, for example) could well cause problems. Similarly, we know that the written requirements document will be incomplete and will not identify all of those potentially troublesome interactions for us. As thoughtful test designers, we want to be smart and systematic about testing for potential problems caused by interactions without going off the deep end and trying to test every possible combination.

Hexawise makes it quick and simple for us to select an appropriate set of tests whatever time pressure might exist on the project or whatever testing thoroughness requirements we might have. Hexawise-generated tests automatically maximize variation, maximize testing thoroughness, and minimize wasteful repetition.

What interesting Hexawise features are highlighted in this sample plan description?

This sample plan write up includes descriptions of the following features:

Married Pairs - How to prevent certain combinations from appearing in your set of tests, faster

Requirements - How to force certain high priority scenarios to appear in your set of tests

Auto-Scripting - How to save time by generating detailed test scripts in the precise format you require (semi)-automatically

Coverage Charts - How to get fact-based insights into "how much testing is enough?"

Matrix Charts - How to tell which exact coverage gaps would exist in our testin gif we were to stop executing tests at any point in time before the final Hexawise-generated test

Using Hexawise's "Coverage Dial" - How to generate sets of thorough 2-way tests and/or extremely thorough 3-way tests in seconds

What interesting test design considerations are raised in this particular sample plan?

However, we might have multiple values that are invalid with one particular value. If that is the case, marking many, many invalid pairs can be time consuming (and frustrating). Hexawise has a shortcut with its Married Pair feature.

In this plan, we might have a scenario where the applicant does not have a spouse. If that is the case, we want to ensure that the scenario does not include an age for a spouse that does not exist. Based on discussions about what age ranges would be necessary for testing, we have 3 particular ranges for spouses. In order to have scenarios that do not include a spouse, we add a fourth value, 'N/A' to the Age of Spouse parameter. Now, we could invalidate the three options of ages with not adding a spouse, or we could use the Married Pair feature.

For this case, we create a Bi-Directional Married Pair between 'Add Spouse? - Do Not Click on Add Spouse' and 'Age of Spouse - N/A.' A Bi-Driectional Married Pair ensures that every test that has 'Add Spouse? - Do Not Click on Add Spouse' will always be paired with 'Age of Spouse - N/A' and every test that has 'Age of Spouse - N/A' will be paired with 'Add Spouse? - Do Not Click on Add Spouse.'

You will also note that when there is a scenario of '0' children added in the insurance quotation, we have should not have average ages of the children present in our scenario. So, similarly, we marry the values of 'How Many Children - 0' with 'Average Age of Children - N/A.'

For more on creating Married Pairs, please see the help file: How do I prevent certain combinations from appearing using the "Married Pair" feature?

Sometimes it is necessary to identify a particular combination as impossible to even test for and remove it from the test cases that Hexawise creates. If you are using a Windows 7 machine, for example, it is not possible to use any version of the browser Safari. For that particular case, we would use the Invalid Pair feature.

Consider additional variation ideas to ask about our verb and noun using "newspaper questions" - who, what, when, why, where, how, how many?

Designing powerful software tests requires that people to think carefully about potential inputs into the system being tested. And how those potential inputs might impact the behavior of the system. As described in this blog post, we strongly encourage test designers to start with a verb an a noun to frame a sensible scope for a set of tests and then ask the "newspaper reporter" questions of who?, what? when? where? why? how? and how many?

Who

Who is asking for the insurance quote and what characteristics do they have that will impact how the System Under Test behaves? In particular...

  • Regarding the applicant's age, how old are they?
  • Regarding the applicant, what gender are they? Do they have to tell us?
  • Does having a family matter?

Who else

  • Does the applicant have a spouse?
  • How old is the spouse, if they have one?
  • Do they have any children?
  • How old are the children, if they have any?

Where

  • Regarding the applicant's location, where are they located?
  • Regarding the applicant's location, what constitues a valid location?
  • Where are they requesting the quote? (e.g., online, on the phone, in person, etc.)

What kind

  • Do they want dental coverage?
  • Do they want maternity coverage?

When

  • What time of day do they get the quote?

Parameters and values entered into Hexawise's Define Inputs screen

Asking the newspaper questions described above is useful to understand potential ways the system under test might behave.

Once we have decided which inputs are important enough to include in this model (and excluded things - like "What time of day do they get the quote?" in this example - that will not impact how the system being tested operates), Hexawise makes it quick and easy to systematically create powerful tests that will allow us to maximize our test execution efficiency.

Once we enter our test inputs into Hexawise, we simply click on the "Create Tests" button at the top of the screen.

Hexawise helps us identify a set of high priority scenarios within seconds


The coverage achieved in the 24 tests above is known as pairwise testing coverage (or 2-way interaction coverage). Hexawise-generated pairwise tests have been proven in many contexts and types of testing to deliver large thoroughness and efficiency benefits compared to sets of hand-selected scenarios.

Hexawise gives test designers control over how thorough they want their testing coverage to be. As in this case, Hexawise allows testers to quickly generate dozens, hundreds, or thousands of tests using Hexawise's "coverage dial." If you have very little time for test execution, you would find those 24 pairwise tests to be dramatically more thorough than a similar number of tests you might select by hand. If you had a lot more time for testing, you could quickly generate a set of even more thorough 3-way tests (as shown in the screen shot immediately below).

For more detailed explanations describing the approach Hexawise uses to maximize variation, maximize coverage, and minimize wasteful repetition in test sets, please see this image-heavy introductory presentation, this 3-page article on Combinatorial Testing (published by IEEE), and/or this detailed explanation comparing the differences between 2-way coverage and 3-way coverage.

Selecting "3-way interactions" generates a longer set of tests which cover every single possible "triplet" of Values

Hexawise generates and displays this extremely thorough set of 90 three-way tests to you within a few seconds.  This set of 3-way coverage strength tests would be dramatically more thorough than typical sets of manually selected test scenarios typically used by large global firms when they test their systems.

The only defects that could sneak by this set of tests would be these two kinds:

  • 1st type - Defects that were triggered by things not included in your test inputs at all (e.g., if special business rules should be applied to an applicant living in Syria, that business rule would not be tested because that test input was never included in the test model at all). This risk is always present every time you design software tests, whether or not you use Hexawise. This risk is, in our experience much larger than the second type of risk:
  • 2nd type - Extraordinarily unusual defects that would be triggered if and only if 4 or more specific test conditions all appeared together in the same scenario. E.g., if the only way a defect occurred was if an applicant (i) is 65 years old and (ii) has a Spouse, lived in (iii) a Rural zip code, and wanted (iv) Maternity coverage. It is extremely rare for defects to require 4 or more specific test inputs to appear together. Many testers test software for years without seeing such a defect. More details are available here.

If a tester spent a few days trying to select tests by hand that achieved 100% coverage of every single possible "triplet" of Values (such as, e.g., (i) 18 year old applicant, and (ii) Without a Spouse, and (iii) has more than 6 children) the following results would probably occur:

  • It would take far longer for a tester to attempt to select a similarly thorough set of tests and the tester would accidentally leave many, many coverage gaps
  • The tester trying to select tests by hand to match this extremely high "all triples" thoroughness level would create far more than 90 tests (which is the optimized solution, shown above)
  • Almost certainly, if the tester tried to achieve this coverage goal in 100 or fewer tests, there would be many, many gaps in coverage (e.g., 3-way combinations of Values that the tester accidentally forgot to include)
  • Finally, unlike the Hexawise-generated tests which systematically minimize wasteful repetition, many of the tester's hand-selected scenarios would probably be highly repetitive from one test to the next; that wasteful repetition would result in lots of wasted effort in the test execution phase

We can force specific scenarios to appear in tests and/or prevent "impossible to test for" combinations from appearing

We easily forced a few high priority scenarios to appear by using Hexawise's "Requirements" feature:

You'll notice from the screen shots of 2-way tests and 3-way tests shown above that some of the Values in both sets of tests are bolded. Those bolded Values are the Values we "forced" Hexawise to include by using this feature.

Auto-scripting allows us to almost instantly convert tables of optimized test conditions (shown above on the "Create Tests" tab screen shots) into detailed test scripts (shown below in the screen shot of an Excel file)

The Auto-scripting feature saves testers a lot of time by partially automating the process of documenting detailed, stepped-out test scripts.

We document a single test script in detail from the beginning to end. As we do so, we indicate where our variables (such as, "Age of Primary Applicant," and "County," and "Gender") are in each sentence. That's it. As soon as we document a single test in this way, we're ready to export every one of our tests.

From there, Hexawise automatically modifies the single template test script we create and inserts the appropriate Values into every test in your plan (whether our plan has 10 tests or 1,000).

We can even add simple Expected Results to our detailed test scripts

If you describe Expected Results like the one above on the "Auto-Scripts" screen, Hexawise will automatically add Expected Results into every applicable test step in every applicable test in your plan. As we entered this Expected Result, every test in this plan will show this Expected Result after test step 14.

It is possible to create simple rules using the drop down menu that will determine when a given Expected Result should appear. To do so, we would use the drop down menus in this feature to create simple rules such as "When ____ is ___ and when ____ is not ____, then the Expected Result would be_____."

This Expected Results feature makes it easy to maintain test sets over time because rules-based Expected Results in tests will automatically update and adjust as test sets get changed over time.

Coverage charts allow teams to make fact-based decisions about "how much testing is enough?"

After executing the first 9 tests of this plan's 2-way set of tests, 77.5% of all possible "pairs" of Values that exist within the system will have been tested together. After all 24 tests, every possible "pair" of Values in the system will have been tested together (100% coverage).

This chart, and the additional charts shown below, provide teams with insights about "how much testing is enough?" And they clearly show that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of test sets is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. As explained here, this type of "diminishing marginal return" is very often the case with scientifically optimized test sets such as these.

Hexawise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in Hexawise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.

We know we would achieve 77.5% coverage of the pairs in the system if we stopped testing after test number 9, but which specific coverage gaps would exist at that point? See the matrix chart below for that information.

The matrix coverage chart tells us exactly which coverage gaps would exist if we stopped executing test before the end of the test set

The matrix chart above shows every specific pair of values that would not yet tested together if we were to stop testing after test number 10.

For example, in the first 10 tests, there is no scenario that includes both (a) "Age of Primary Applicant - 40 - 64.9" together with (b) "Gender - Male."

You may notice that there are several black boxes in the Matrix Chart above. Those black boxes represent the invalid pairs created via the Invalid Pairs or Married Pairs features. We mark them as black to indicate that Hexawise will not pair them together, whereas the red boxes are those pairs that have yet to be covered.

We can analyze coverage on the extremely thorough set of 3-way tests we created also.

After executing the first 30 tests of this plan's 3-way set of tests, 80.4% of all possible "triplets" of Values that exist within the system will have been tested together.  After all 90 tests, every possible "triplet" of Values in the system will have been tested together (100% coverage).

This chart provides teams with insights about "how much testing is enough?" And it clearly shows that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of the test set is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. As explained here, this type of "diminishing marginal return" is very often the case with scientifically optimized test sets such as these.

Hexawise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in Hexawise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.

Mind maps can be exported from this Hexawise plan to facilitate stakeholder discussions.

Hexawise supports exporting in several different formats. Mind maps can be a great option if a tester wants to get quick, actionable guidance from stakeholders about which test inputs should (or should not) be included. Mind maps quickly demonstrates to stakeholders that the test designers have thought about the testing objectives clearly and they give stakeholders an opportunity to provide useful feedback more quickly as compared to having stakeholders read through long documents filled with test scripts.

Detailed test scripts (complete with stepped-out tester instructions and rule-generated Expected Results) can be exported also:

The detailed test scripts shown above were created using Hexawise's Auto-Scripts feature.

Other possible export formats could include test "data tables" in either CSV or Excel format or even Gherkin-style formatting.

At Hexawise, we regularly customize export formats to exactly match our client's specific formatting requirements. This can make exporting from Hexawise and importing into your customized version of HP QC / HP ALM very quick and easy.

Did this answer your question?