Negative testing, pairwise testing, orthogonal array testing, and combinatorial testing
There are three general types of scenarios that are relevant to the discussion of negative tests in Hexawise.
Treat "impossible-to-test-for" scenarios differently than negative tests
Impossible-to-test-for scenarios involve combinations of test inputs that will NEVER appear together in the real world. A good example would be using an Apple computer's Operating System and trying to launch Internet Explorer. It cannot be done under typical circumstances. There is no point in trying to test for it, because Internet Explorer has not been available on Mac computers for years. The appropriate way to handle these scenarios is to simply prevent them from appearing in your tests using the Invalid Pair or Married Pair feature.
Negative Tests are different. You want to include certain scenarios in order to confirm, for example, that certain types of users will not be able to perform certain actions. If you had different types of users for an airplane reservation system, for example, you might want to confirm that no role other than a Super-Admin User would be able to modify the ticket price. It is important to test for these kinds of scenarios.
It might be confusing at first to understand how to address negative tests in the context of generating Hexawise tests, but the effort you make to clearly understand your negative testing options will be well worth it because this topic comes up frequently in most projects.
Advice for Beginners: Keep negative tests separate from Hexawise tests
The easiest way to handle negative tests is to simply keep them separate from your Hexawise-generated tests. Many teams using Hexawise find it easiest just to use Hexawise to generate their positive tests. Then, outside of Hexawise, they will document negative tests in the same way prior to Hexawise.
There is nothing wrong with this approach and it has the advantage of being "clean" and easy to explain. Depending upon your teams tools and processes, and perhaps the quantity and nature of the particular negative tests you have in mind, it might be an appropriate solution for you.
If you decide to use this approach, consider using the "Notes" feature within Hexawise; some teams use the Notes feature to document their ideas for negative tests to make sure they don't get lost.
Advice for Intermediate / Advanced Users: Distinguish between negative tests that "kill the script," and negative tests that allow testers to execute the complete scenario
Hexawise sets of tests include tests that generally have the same number of test steps from one test to the next. It is important that every test actually gets executed from start to finish. Why? Because of interactive coverage measurement. Hexawise tests ensure that you will achieve coverage of all of the interactions you have selected (2-way interactions, 3-way, etc.) If some of your tests stop part-way through, you will not achieve your desired coverage goal. The following example demonstrates this important point.
Imagine that you have a test that says:
- Fly from: India
- Fly to: France
- Departure Date: Tomorrow
- Class of Travel: First
- Number of Passengers: 500 <<Which causes the test to fail here>>
- Meal Preference: Vegetarian
- Type of Discount Available: XYZ
- Payment Type: CC, Cash, FF Miles
- Print ticket: Online, Email, Snail Mail
What is the problem with this kind of scenario that would risk "Killing the Script" in the 5th test? Steps 6 through 9 would never get executed. The Hexawise coverage algorithm has specially selected each of the values that appear in each of the 9 steps to achieve your coverage objective. But if this test were to get killed after test 5, all of the following interactions that should be tested for in this test would never actually get tested:
From India & Vegetarian
From India & XYZ Discount type
From India & FF Miles
From India & Snail Mail
To France & Vegetarian
To France & XYZ Discount type
To France & FF Miles
To France & Snail Mail
Depart Tomorrow & Vegetarian
Depart Tomorrow& XYZ Discount type
Depart Tomorrow & FF Miles
Depart Tomorrow & Snail Mail
First Class Travel & Vegetarian
First Class Travel & XYZ Discount type
First Class Travel & FF Miles
First Class Travel & Snail Mail
Vegetarian & XYZ Discount Type
Vegetarian & FF Miles
Vegetarian & Snail Mail
XYZ Discount Type & FF Miles
XYZ Discount Type & Snail Mail
FF Miles & Snail Mail
Advice for Intermediate and Advanced Users: Consider including negative tests that allow execution of every step in tests until the end
Sometimes it is simple to avoid the problem described above. Sometimes you can just change the way you're describing the Value that causes an error message to appear in a way that makes it possible to achieve both of these objectives:
- Trigger the error message to complete your negative test, and
- Enable the tester to fix the problem and proceed to the end of the test.
In the flight reservation example above, here's how we could change the description of the Value we enter into Hexawise to accomplish both of those goals:
- Instead of entering "500" as the quantity of passengers, enter the following: "Enter '500 passengers,' then confirm that the correct error message appears, then enter a valid number of passengers."
Beware of negative test ideas that would kill any of your test scripts before the final step. Document those outside of Hexawise.
The example above assumes that a tester would be able to trigger an error message with an invalid entry, and then fix the problem that caused it and continue onwards to execute the entire test script. Sometimes, though, a tester might not be able to fix the problem nor continue executing steps. The following kind of test for an ATM is an example of a "script-killing" test:
"Confirm that when an ATM user enters their PIN incorrectly 5 times, then the machine should physically destroy the user's card."
If you had a set of tests that involved multiple steps and were supposed to happen after the user entered their PIN code, such as withdraw a certain amount of money from a certain type of account, you would run into the problem described immediately above. Important combinations of test inputs would not be tested. In this example, there is nothing that a tester would be able to do after the card was physically destroyed; it would not work to say, "now fix the problem and continue executing the test." It would be impossible to resurrect the card once it had been physically destroyed.
If you have a script-killing test that you want to run, do not include it in your Hexawise tests. Instead, create a test for that scenario outside of Hexawise.