Usability test results
Test Method
Using the test protocol, it was possible to compare the innovative map-based e-consultation technique with the conventional paper questionnaire used in the Probation Board consultation. The comparison is shown in another chart (See Figure 2)
The chart shows the difference between the post-test scores given to the web map and the paper consultation questionnaire. The scores were 5 for strongly agree down to 1 for strongly disagree. The difference can range from +4 to -4. Where the question was a negative one, the difference was taken away from 0, so that in all cases +4 is better for the web map, and -4 is worse.
Test Results
Scores suggest that the the paper questionnaire was considered as better.
Now for two of the testers (R and S) (See Figure 1), the web map was better than the paper questionnaire on all but one criterion. The exception was the organization of information on the page.
For those with less Internet experience, the questionnaire was better. But on every other criterion of ease of use, the map-based interface was better.
The third tester (P) was over 60, and had had less experience of computers and the Internet. On some criteria he found the paper questionnaire easier (based on his experience of filling in forms in the past). Nevertheless, he was more satisfied overall with the new web map than with the traditional questionnaire.
During the tests, there were frequent complaints about the paper questionnaire. Q5 was particularly hard to grasp. It reads like a university or A-level examination question:
5. PBNI Equality Scheme has given a commitment that in carrying out its functions it will have due regard to the need to promote equality of opportunity between people who fall under the following groups as stated in s75 of the Northern Ireland Act 1998. To help us consider potential inequalities please identify the people in the following groups who you believe will be most affected by the proposed changes (for staff and service users):- Religious belief: Positive [ ] Negative [ ] ..
However, even when answering the simpler questions, the testers discussed what the questions mean before attempting to answer.
In contrast, there were few problems when using the on-line map. They found the probation office locations, recognised local features, and managed to manipulate and move around the map display. Writing in several words as a comment on a particular site was not a problem for two of them: the third got the research assistant to help him. In the discussion afterwards, they expressed their satisfaction with the map interface, and how easy it was to use.
The following are some sample comments added to locations on the map.
- Alderwood House (ISU). There is car parking, and bus and shopping. I feel safe to go to office. There is good services offer in office. Some community perception and people might not go over there. —P
- Belfast court (Headquarter). They would be affected by bus strick. They would try to find another way to visit a location—P
- lt is just convienet to people living in belfast to go to both courts beside center.—P
- Ormeau Road There is no disadvantage to this office, staff are very helpful, safe room to speack to probation office.—P
- It is parking for people, easy for bues and shops.—P
- Shankill Road. The marker wasnt near the location. i live on this road and i got a bit confused finding it on the map.The marker should be on the location.—S
- Lisburn Office. found the location i needed. needs no improvements.—S
- Newtownards there needs to be more improvement on the map details. if i was really looking for the location i wouldnt get any where because its confusing.—S
These comments relate directly to the issues of the consultation, focused on specific sites. By starting from the particular, rather than requiring consultees to give general comments, it is both easier for consultees to express their views, and consulters to understand and analyse them.
