This is an overview and explanation of a set of configuration-based functional tests created in the Hexawise test design tool.
What are our testing objectives?
More than 100 billion possible combinations - that's how many there are once you take into account all of the different configuration options and other variables in this sample plan. In this context, we want to test this printing process relatively thoroughly - with a manageable number of tests.
We know that testing each item in our system once is not sufficient; we know that interactions between the different things in our system could well cause problems. Similarly, we know that the written requirements document will be incomplete. As thoughtful test designers, we want to be smart about testing for potential problems caused by interactions without going off the deep end and trying to test every possible combination.
Hexawise makes it quick and simple for us to select an appropriate set of tests whatever time pressure might exist on the project or whatever testing thoroughness requirements we might have. Hexawise-generated tests automatically maximize variation, maximize testing thoroughness, and minimize wasteful repetition.
What interesting Hexawise features are highlighted in this sample plan description?
This sample plan write up includes descriptions of the following features:
Requirements - How to force certain high priority scenarios to appear in your set of tests
Auto-Scripting - How to save time by generating detailed test scripts in the precise format you require (semi)-automatically
Coverage charts - How to get fact-based insights into "how much testing is enough?"
Using Hexawise's "Coverage Dial" - How generate sets of thorough 2-way tests and/or extremely thorough 3-way tests in seconds
What interesting test design considerations are raised in this particular sample plan?
There are a few test design considerations that are notable about this sample plan:
- There are fewer than 100 tests created when 2-way coverage is used - even though there are more than 100 billion potential scenarios
- That's an extraordinary ratio. How did this result happen in this plan? And how might similar results come about in other plans?
- Whenever you have lots of parameters, like here, you wind up with a combinatorial explosion: in this case, with more than 20 different parameters, there are more than 100 billion different potential combinations of test inputs
- Hexawise is helpful when you have hundreds of possible tests, very helpful when you have thousands of possible tests and extremely helpful in situations, like here, where you cannot realistically test all the different combinations you could think of
This kind of result (e.g., only a few dozen tests are required to achieve 2-way / pairwise coverage even though there are hundreds of billions of possible tests) only happens in situations where both of the following things are true:
- There are many (in this case, slightly more than 20) Parameters, and
- Each of the Parameters has a small number of Values (in this case, there are 9 Values for File Types and 7 Values for Printer Manufacturer)
It is also worth highlighting that no matter whether you want ot execute a set of a few dozen, a few hundred, or a few thousand tests, Hexawise can generate a prioritized / optimized set of tests for your thoroughness and timing needs
Consider additional variation ideas to ask about our verb and noun using "newspaper questions" - who, what, when, why, where, how, how many?
Designing powerful software tests requires people to think carefully about potential inputs into the system being tested. As described in this blog post, we strongly encourage test designers to start with a verb an a noun to frame a sensible scope for a set of tests and then ask the "newspaper reporter" questions of who?, what? when? where? why? how? and how many?
- What type of device will be used to send the printing instructions?
- What operating system will be on the device?
- What browser type will be on the device?
- What type of file will be printed?
- What kind of text will be included in the document?
- What document formatting options will be used in the file? (margin sizes, justification, etc.)
- What additional elements will be included in the file? (with and without charts, images, etc.)
How Many / How Large
- How large will the file be?
- Which sized paper will be used?
- How will the printing instructions be sent? (this system allows for printing directly via wi-fi or printing indirectly via email)
- How is the printing instruction routed if it's being sent by email? / via which type of mail server?
- How is the printing instruction routed if it's being sent by wi-fi?
- Which type of printer will be used?
- Will the document be collated?
- Will multiple pages be printed on each seat?
- Which tray will be used?
Parameters and values entered into Hexawise's Define Inputs screen
Asking the "newspaper reporter" questions described above helps us to identify many possible scenarios that could potentially impact how the system behaves. Once we have decided what inputs are important enough to include in your test inputs (and probably excluded things that will not impact how the system being tested operates such as "color of paper being printed" in this example), Hexawise makes it quick and easy to systematically create powerful tests that will allow us to maximize our test execution efficiency.
Once we enter our test inputs into Hexawise, we simply click on the "Create Tests" button at the top of the screen.
Hexawise helps us identify a set of high priority scenarios within seconds
The coverage achieved in the 72 tests above is known as pairwise testing coverage (or 2-way interaction coverage). Hexawise-generated pairwise tests have been proven in many contexts and types of testingto deliver large thoroughness and efficiency benefits compared to sets of hand-selected scenarios.
Hexawise gives test designers control over how thorough they want their testing coverage to be. As in this case, Hexawise allows testers to quickly generate dozens, hundreds, or thousands of tests using Hexawise's "coverage dial." If you have very little time for test execution, you would find those 72 pairwise tests to be dramatically more thorough than a similar number of tests you might select by hand. If you had a lot more time for testing, you could quickly generate a set of even more thorough 3-way tests (as shown in the screen shot immediately below).
For more detailed explanations describing the approach Hexawise uses to maximize variation, maximize coverage, and minimize wasteful repetition in test sets, please see this image-heavy introductory presentation, this 3-page article on Combinatorial Testing (published by IEEE), and/or this detailed explanation comparing the differences between 2-way coverage and 3-way coverage.
Selecting "3-way interactions" generates a longer set of tests which cover every single possible "triplet" of Values
Hexawise generates and displays this extremely thorough set of 536 3-way tests to you within a few seconds. This set of 3-way coverage strength tests would be dramatically more thorough than typical sets of manually selected test scenarios typically used by large global firms when they test their systems.
If a tester spent a few days trying to select tests by hand that achieved 100% coverage of every single possible "triplet" of Values (such as, e.g., (i) Using a PC, (ii) printing an Adobe PDF file, and (iii) with an IMAP mail server" the following results would probably occur:
- It would take far longer for a tester to try to select a similarly thorough set of tests and the tester would accidentally leave many, many coverage gaps
- The tester trying to select tests by hand to match this extremely high "all triples" thoroughness level would create far more than 536 tests (which is the optimized solution, shown above)
- Almost certainly, if the tester tried to achieve this coverage goal in 600 or fewer tests, there would be many, many gaps in coverage (e.g., 3-way combinations of Values that the tester accidentally forgot to include)
- Finally, unlike the Hexawise-generated tests which systematically minimize wasteful repetition, many of the tester's hand-selected scenarios would probably be highly repetitive from one test to the next; that wasteful repetition would result in lots of wasted effort in the test execution phase
We can force specific scenarios to appear in tests and/or prevent "impossible to test for" combinations from appearing
We easily forced a few high priority scenarios to appear by using Hexawise's "Requirements" feature:
You'll notice from the screen shots of 2-way tests and 3-way tests shown above that some of the Values in both sets of tests are bolded. Those bolded Values are the Values we "forced" Hexawise to include with this feature.
Auto-scripting allows us to almost instantly convert tables of optimized test conditions (shown above on the "Create Tests" tab screen shots) into detailed test scripts (shown below in the screen shot of an Excel file)
The Auto-scripting feature saves testers a lot of time by partially automating the process of documenting detailed, stepped-out test scripts.
We document a single test script in detail from the beginning to end. As we do so, we indicate where our variables (such as, "File type," and "Printer Manufacturer," and "Pages per Sheet") are in each sentence. That's it. As soon as we document a single test in this way, we're ready to export every one of our tests.
From there, Hexawise automatically modifies the single template test script we create and inserts the appropriate Values into every test in your plan (whether our plan has 10 tests or 1,000).
We can even add simple Expected Results to our detailed test scripts
If you describe Expected Results like the one above on the "Auto-Scripts" screen, Hexawise will automatically add Expected Results into every applicable test step in every applicable test in your plan. As we entered this Expected Result, every test in this plan will show this Expected Result after test step 6.
It is possible to create simple rules using the drop down menu that will determine when a given Expected Result should appear. To do so, we would use the drop down menus in this feature to create simple rules such as "When ____ is ___ and when ____ is not ____, then the Expected Result would be_____."
This Expected Results feature makes it easy to maintain test sets over time because rules-based Expected Results in tests will automatically update and adjust as test sets get changed over time.
Coverage charts allow teams to make fact-based decisions about "how much testing is enough?"
After executing the first 40 tests of this plan's 2-way set of tests, 90.9% of all possible pairs of Values that appear in this plan will have been tested together.
This chart provides teams with insights about "how much testing is enough?" And it clearly shows that the amount of learning / amount of coverage that would be gained from executing the tests at the beginning of the test set is much higher than the the learning and coverage gained by executing those tests toward the end of the test set. This is very often the case with scientifically optimized test sets such as these.
Hexawise tests are always ordered to maximize the testing coverage achieved in however much time there is available to test. Testers should generally execute the tests in the order that they are listed in Hexawise; doing this allows testers to stop testing after any test with the confidence that they have covered as much as possible in the time allowed.
Mind maps can be exported from this Hexawise plan to facilitate stakeholder discussions.
Hexawise supports exporting in several different formats. Mind maps can be a great option if a tester wants to get quick, actionable guidance from stakeholders about which test inputs should (or should not) be included. Mind maps quickly demonstrates to stakeholders that the test designers have thought about the testing objectives clearly and they give stakeholders an opportunity to provide useful feedback more quickly as compared to having stakeholders read through long documents filled with test scripts.
Detailed test scripts (complete with stepped-out tester instructions and rule-generated Expected Results) can be exported also:
The detailed test scripts shown above were created using Hexawise's Auto-Scripts feature.
At Hexawise, we regularly customize export formats to exactly match our client's specific formatting requirements. This can make exporting from Hexawise and importing into your customized version of HP QC / HP ALM very quick and easy.