|
Evaluation | Choosing the Test
You must consider the type of information you want to get from the test, and the time required to analyze it and reach conclusions. The quicker the test can be analyzed, the quicker you can incorporate your conclusions into the R&D process.
The following table sums up the different types of test:
The following table sums up the different types of test:
Type of test
Short description
Advantages
Disadvantages
A small number of testers (e.g. 5) perform a number of pre-defined actions and while doing so, voice their thoughts and decision processes. The project team sits beside them and writes down their conclusions.
The testers may be either company employees or outsiders selected according to the required criteria. It is easier and cheaper to use the former, but they may not match the required user profiles.
The testers may be either company employees or outsiders selected according to the required criteria. It is easier and cheaper to use the former, but they may not match the required user profiles.
Quick and cheap; does not require special equipment; flexible – no need to keep to the predefined protocol: questions that arise during the testing process can also be explored.
the testers are not always comfortable talking aloud while using the application.
User interviews
A small number (10 – 20) users are interviewed by phone in order to get feedback regarding different aspects of the user experience and any problems or difficulties with it.
Quick and cheap; well-defined and accessible user segments
May fail to identify the problems of less experienced users; will identify only major problems which users remember, not everything that detracts from the user experience during actual use.
User surveys
A representative sample (e.g. of several dozen users) is requested to perform a number of defined operations and fill in a questionnaire about the process. The users report on their own, without the presence of anyone from the team.
Allows a relatively large user sample; different user populations can be sampled, allowing the identification of unusual or esoteric problems.
Subjective report, not from a trained tester; users who did not manage to perform the requested tasks may not report back at all.
Cognitive walkthrough
Groups each consisting of a number of users (e.g. 5) plus some of the project team perform an orderly “walk through” of different components of the product and point out any problems or difficulties with each component. This is a group process, with all members of the group allowed enough time to check out the product and perform the requested tasks.
Simple and easy; provides meaningful feedback plus new ideas for improving the product. Can be performed as an addition to the Think-aloud Protocol.
Success depends on the intra-group dynamics; excessive dominance by the project team members, or one/some of them, may prevent one or more users from providing their feedback.
Mouse-tracking tests
Tools such as provided by http://www.clickdensity.com/ which collect and analyze a website user’s mouse movements and identify problem areas.
identifies problematic or confusing areas; relatively few (e.g. 1000) users are needed; can be turned on and off.
may slow down website response time; must send information to the external website of the testing company.
Optimization tests
Tools such as Google Website Optimizer
allows improvement of the user experience for specific web pages, by checking the different components of the page.
users must perform the analysis via an external company’s server; requires considerable proficiency in carrying out the test; several alternative versions of each page to be tested must be prepared.