This is an overview and explanation of a set of configuration-based functional tests created in the Hexawise test design tool.

What are our testing objectives?

More than 100 trillion possible combinations - that's how many there are once you take into account all of the different configuration options and other test design considerations in this sample plan. In this context, we want to test the various configurations available - with a manageable number of tests.

We know that testing each item in our system once is not sufficient; we know that interactions between the different configuration settings in our system could well cause problems. Similarly, we know that the written requirements document will be incomplete because, realistically, it does not identify all of the potentially troublesome interactions. As thoughtful test designers, we want to be smart about testing for potential problems caused by interactions without going off the deep end and trying to test every possible combination.

Hexawise makes it quick and simple for us to select an appropriate set of tests whatever time pressure might exist on the project or whatever testing thoroughness requirements we might have. Hexawise-generated tests automatically maximize variation, maximize testing thoroughness, and minimize wasteful repetition.

What interesting Hexawise features are highlighted in this sample plan description?

This sample plan write up includes descriptions of the following features:

Weighting - How to have a value appear more often in your set of tests

Requirements - How to force certain high priority scenarios to appear in your set of tests

Auto-Scripting - How to save time by generating detailed test scripts in the precise format you require (semi)-automatically

Coverage charts - How to get fact-based insights into "how much testing is enough?"

Using Hexawise's "Coverage Dial" - How generate sets of thorough 2-way tests and/or extremely thorough 3-way tests in seconds

What interesting test design considerations are raised in this particular sample plan?

There are a few test design considerations that are notable about this sample plan:

First, the parameters and values are quite easy to identify. This Java configuration test plan uses the Java preferences available on an Apple Macbook, which are all nicely laid out across a handful of screens located under System Preferences. Normally, a system under test requires a good deal of mental effort to identify the necessary parameters and values. What makes this even easier is the binary nature of most of the options (Enabled, Not enabled).

Second, we weight the values to create tests that, we consider, to be more 'realistic.' This is clear in the number of times "Do not select' appears in any given parameter. For parameters where the option to enable or not enable a particular Java preference, we choose to have two (2) 'Do not select' for every 'Select.' This does increase our overall number of tests (by 7 tests), but the benefit of having more realistic tests outweighs that cost.

Third, the resulting number of tests is quite small even though there are a lot of parameters present in this sample plan.

  • There are fewer than 50 tests created when 2-way coverage is used - even though there are more than 100 trillion potential scenarios
  • That's an extraordinary ratio. How did this result happen in this plan? And how might similar results come about in other plans?
  • Whenever you have lots of parameters, like here, you wind up with a combinatorial explosion: in this case, with more than 20 different parameters, there are more than 100 billion different potential combinations of test inputs
  • Hexawise is helpful when you have hundreds of possible tests, very helpful when you have thousands of possible tests and extremely helpful in situations, like here, where you cannot realistically test all the different combinations you could think of

This kind of result (e.g., only a few dozen tests are required to achieve 2-way / pairwise coverage even though there are hundreds of billions of possible tests) only happens in situations where both of the following things are true:

  • There are many (in this case, slightly more than 20) Parameters, and
  • Each of the Parameters has a small number of Values (in this case, there are 9 Values for File Types and 7 Values for Printer Manufacturer)

It is also worth highlighting that no matter whether you want ot execute a set of a few dozen, a few hundred, or a few thousand tests, Hexawise can generate a prioritized / optimized set of tests for your thoroughness and timing needs

Consider additional variation ideas to ask about our verb and noun using "newspaper questions"

Designing powerful software tests requires people to think carefully about potential inputs into the system being tested. As described in this blog post, we strongly encourage test designers to start with a verb an a noun to frame a sensible scope for a set of tests and then ask the "newspaper reporter" questions of who?, what? when? where? why? how? and how many?

To be clear, this is represented as an exercise, but clearly many, if not all, of the parameters and values could be discovered by viewing the Java preferences screens.

What

  • What network connections will be used?
  • What level of compression for JAR files should be used?
  • What tracing rights should be established for Debugging purposes?
  • What logging rights should be established for Debugging purposes?
  • What lifecycle exception rights should be established for Debugging purposes?
  • What right should the user have to grant permission to signed content?
  • What right should the user have to grant permission to untrusted authority?
  • What should happen if the site certificate and hostname do not match?
  • What permission does the user have for accepting JNLP security requests?
  • What status is the trusted publisher list given?
  • What SSL formats are enabled?

How Many / How Large

  • How large should the temporary file storage be?
  • How often should revocation certificates be checked?

How

  • How should cache files be handled? (viewed, not viewed)
  • How should any and all files post-testing be handled? (delete, keep)
  • How should Java test input method keys be handled?
  • How should the Java console be shown?
  • How should certificates and keys be handled in browser?
  • How should be sandbox warnings be handled?
  • How can blacklist revocation be checked?
  • How do warnings on mixed security codes get processed?

Where

  • Where will the applets be run? (in their own process, within the browser process)
  • Where will the automatic proxy script be stored?
  • Where should temporary files be stored?
  • Where should certificate validations be performed?

When

  • When during the day should the preferences be set?
  • When should shortcuts be allowed to be created?
  • When should prompts for client certificates be allowed?
  • When should the site certificate be shown?
  • When should password caching be allowed?

Parameters and values entered into Hexawise's Define Inputs screen

Asking the "newspaper reporter" questions described above helps us to identify many possible scenarios that could potentially impact how the system  behaves. Once we have decided what inputs are important enough to include in your test inputs (and probably excluded things that will not impact how the system being tested operates such as "time of day for setting preferences" in this example), Hexawise makes it quick and easy to systematically create powerful tests that will allow us to maximize our test execution efficiency.

Once we enter our test inputs into Hexawise, we simply click on the "Create Tests" button at the top of the screen.

Hexawise helps us identify a set of high priority scenarios within seconds

The coverage achieved in the 31 tests above is known as pairwise testing coverage (or 2-way interaction coverage). Hexawise-generated pairwise tests have been proven in many contexts and types of testingto deliver large thoroughness and efficiency benefits compared to sets of hand-selected scenarios.

Hexawise gives test designers control over how thorough they want their testing coverage to be. As in this case, Hexawise allows testers to quickly generate dozens, hundreds, or thousands of tests using Hexawise's "coverage dial." If you have very little time for test execution, you would find those 31 pairwise tests to be dramatically more thorough than a similar number of tests you might select by hand. If you had a lot more time for testing, you could quickly generate a set of even more thorough 3-way tests (as shown in the screen shot immediately below).

For more detailed explanations describing the approach Hexawise uses to maximize variation, maximize coverage, and minimize wasteful repetition in test sets, please see this image-heavy introductory presentation, this 3-page article on Combinatorial Testing (published by IEEE), and/or this detailed explanation comparing the differences between 2-way coverage and 3-way coverage.

Selecting "3-way interactions" generates a longer set of tests which cover every single possible "triplet" of Values

Hexawise generates and displays this extremely thorough set of 167 3-way tests to you within a few seconds.  This set of 3-way coverage strength tests would be dramatically more thorough than typical sets of manually selected test scenarios typically used by large global firms when they test their systems.

If a tester spent a few days trying to select tests by hand that achieved 100% coverage of every single possible "triplet" of Values (such as, e.g., (i) running applets in their own process, (ii) while the Java console is hidden, and (iii) online certificate validations are not enabled) the following results would probably occur:

  • It would take far longer for a tester to try to select a similarly thorough set of tests and the tester would accidentally leave many, many coverage gaps
  • The tester trying to select tests by hand to match this extremely high "all triples" thoroughness level would create far more than 166 tests (which is the optimized solution, shown above)
  • Almost certainly, if the tester tried to achieve this coverage goal in 200 or fewer tests, there would be many, many gaps in coverage (e.g., 3-way combinations of Values that the tester accidentally forgot to include)
  • Finally, unlike the Hexawise-generated tests which systematically minimize wasteful repetition, many of the tester's hand-selected scenarios would probably be highly repetitive from one test to the next; that wasteful repetition would result in lots of wasted effort in the test execution phase

We can force specific scenarios to appear in tests

We easily forced a few high priority scenarios to appear by using Hexawise's "Requirements" feature:

You'll notice from the screen shots of 2-way tests and 3-way tests shown above that some of the Values in both sets of tests are bolded. Those bolded Values are the Values we "forced" Hexawise to include with this feature.

Auto-scripting allows us to almost instantly convert tables of optimized test conditions into detailed test scripts

The Auto-scripting feature saves testers a lot of time by partially automating the process of documenting detailed, stepped-out test scripts.

We document a single test script in detail from the beginning to end. As we do so, we indicate where our variables (such as, "Network Settings," and "Set Key," and "Enable online certificate validation") are in each sentence. That's it. As soon as we document a single test in this way, we're ready to export every one of our tests.

From there, Hexawise automatically modifies the single template test script we create and inserts the appropriate Values for every test in your plan (whether our plan has 10 tests or 1,000).

We can even add simple Expected Results to our detailed test scripts

If you describe Expected Results like the one above on the "Auto-Scripts" screen, Hexawise will automatically add Expected Results into every applicable test step in every applicable test in your plan. As we entered this Expected Result, every test in this plan will show this Expected Result after test step 4.

It is possible to create simple rules using the drop down menu that will determine when a given Expected Result should appear. To do so, we would use the drop down menus in this feature to create simple rules such as "When (a particular parameter) is (a particular value), then the Expected Result would be_____."

This Expected Results feature makes it easy to maintain test sets over time because rules-based Expected Results in tests will automatically update and adjust as test sets get changed over time.

Coverage charts allow teams to make fact-based decisions about "how much testing is enough?"

After executing the first 10 tests of this plan's 2-way set of tests, 79.2% of all possible "pairs" of Values that exist within the system will have been tested together. After all 32 tests, every possible "pair" of Values in the system will have been tested together (100% coverage).

This chart, and the additional charts shown below, provide teams with insights about "how much testing is enough?" And they clearly show that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of test sets is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. As explained here, this type of "diminishing marginal return" is very often the case with scientifically optimized test sets such as these.

Hexawise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in Hexawise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.

We know we would achieve 79.2% coverage of the pairs in the system if we stopped testing after test number 10, but which specific coverage gaps would exist at that point? See the matrix chart below for that information.

The matrix coverage chart tells us exactly which coverage gaps would exist if we stopped executing test before the end of the test set

The matrix chart above shows every specific pair of values that would not yet tested together if we were to stop testing after test number 10.

For example, in the first 10 tests, there is no scenario that includes both (a) Network Settings - Use system settings together with (b) Use automatic proxy configuration script - invalid script location.

Hexawise-generated tests will fill as many coverage gaps as possible with each test. In contrast, hand-selected tests that testers might select themselves would leave far more gaps in coverage.

We can analyze coverage on the extremely thorough set of 3-way tests we created also.

After executing the first 38 tests of this plan's 3-way set of tests, 80.2% of all possible "triplets" of Values that exist within the system will have been tested together. After all 166 tests, every possible triplet of Values in the system will have been tested together (100% coverage).

This chart provides teams with insights about how much testing is enough? And it clearly shows that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of the test set is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. As explained here, this type of diminishing marginal return is very often the case with scientifically optimized test sets such as these.

Hexawise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in Hexawise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.

Mind maps can be exported from this Hexawise plan to facilitate stakeholder discussions.

Hexawise supports exporting in several different formats. Mind maps can be a great option if a tester wants to get quick, actionable guidance from stakeholders about which test inputs should (or should not) be included. Mind maps quickly demonstrates to stakeholders that the test designers have thought about the testing objectives clearly and they give stakeholders an opportunity to provide useful feedback more quickly as compared to having stakeholders read through long documents filled with test scripts.

Detailed test scripts (complete with stepped-out tester instructions and rule-generated Expected Results) can be exported also:

The detailed test scripts shown above were created using Hexawise's Auto-Scripts feature.

Other possible export formats could include test "data tables" in either CSV or Excel format or even Gherkin-style formatting.

At Hexawise, we regularly customize export formats to exactly match our client's specific formatting requirements. This can make exporting from Hexawise and importing into your customized version of HP QC / HP ALM very quick and easy.

Did this answer your question?