There are two common problems with the use of the pair test. Both problems are frequently overlooked by the users. First, the preferred choice in a pair test, does not reflect market success. It does not even indicate which of the two has a better chance in the marketplace. "Discrimination does not equal importance." The market configuration will determine success. As an example, if we are testing a green vs. blue shampoo, and color is a dominating factor, the blue shampoo may be significantly preferred to the green shampoo. But if the market has multiple blue shampoo brands and no green shampoos, the introduction of the blue candidate will split shares among the other blue brands, while the green shampoo will capture the green population. L. L. Thurstone wrote volumes on this subject in the early 1930's.
The second problem deals with carry-over effects. In a single pair test, the carry-over effect is seen as an order effect while in a round robin analysis, it is usually reflected as a product X order interaction. Basically, this effect says that the results are determined by the order of use. For example, if we are testing a laundry detergent against a laundry detergent with an additional active such as bleach or fabric softener, we would expect to see misleading pair results (significant order effect). The panelists using the regular detergent first and the detergent with bleach second will report a substantial win for the detergent + bleach, something like an 80/20 win. However, among the panelists using the products in the opposite order, a smaller difference will be reported because the detergent + bleach will clean up much of the redeposition and stain problems so that when the regular detergent is used, there will be very little visual differences in the results (55/45). The reported result would be a 68/32 when the 80/20 is probably closer to the truth.
For further information on this topic go to www.popsg.org/views.