Difference between revisions of "Designing a usability test"

Revision as of 18:28, 13 January 2008

We set out to compare the usability of this map-based interface with the conventional paper questionnaire produced by the PBNI.

Selecting the testers

There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques. We needed to recruit ex-offenders to test the system. The Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO) runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So NIACRO arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006.

Selecting the tasks

We designed a sequence of tasks that started with simple familiarization with the map-based site, then got progressively more involved as the testers gained confidence, until they were entering comments on the consultation topic. In brief they were: · Explore the PBNI e-consultation website. · Find your probation office or reporting centre on the map. · Find the all offices or reporting centres you have been to, and add comments on each place. · Find your new office and reporting centre, and comment on how it would affect you. The control task was to complete as much of the PBNI questionnaire as they could in 15 minutes.

Collecting test data

Before the tests started, we asked testers to complete a questionnaire on their familiarity with computers, the Internet and consultations. We set up Camtasia Studio on one PC. This software produced a video of the screen, including the position of the mouse at any time, and what the tester saw. The tester spoke aloud during the test, explaining what s/he was doing, and noting any difficulties in using the interface. This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server. After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire. We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire. Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised.