Difference between revisions of "Usability test results"
| Line 1: | Line 1: | ||
==Test Method== | ==Test Method== | ||
Using the test protocol, it was possible to compare the innovative map-based e-consultation | Using the test protocol, it was possible to compare the innovative map-based e-consultation | ||
| − | technique with the conventional paper questionnaire used in the [http://www.pbni.org.uk/ Probation Board] consultation. The comparison is shown in | + | technique with the conventional paper questionnaire used in the [http://www.pbni.org.uk/ Probation Board] consultation. The comparison is shown in another chart (See Figure 2) |
[[Image:PBNI_Consult_Chart.png|thumb|Figure 1: Chart]] | [[Image:PBNI_Consult_Chart.png|thumb|Figure 1: Chart]] | ||
| + | |||
| + | [[Image:Web_map_test_results.png|thumb|Figure 2: Web map vs. Paper Questionnaires]] | ||
The chart shows the difference between the post-test scores given to the web map and the paper consultation questionnaire. The scores were 5 for strongly agree down to 1 for strongly disagree. The difference can range from +4 to -4. Where the question was a negative one, the difference was taken away from 0, so that in all cases +4 is better for the web map, and -4 is worse. | The chart shows the difference between the post-test scores given to the web map and the paper consultation questionnaire. The scores were 5 for strongly agree down to 1 for strongly disagree. The difference can range from +4 to -4. Where the question was a negative one, the difference was taken away from 0, so that in all cases +4 is better for the web map, and -4 is worse. | ||
| Line 10: | Line 12: | ||
Scores suggest that the the paper questionnaire was considered as better. | Scores suggest that the the paper questionnaire was considered as better. | ||
| − | Now for two of the testers (R and S), the web map was better than the paper questionnaire on all | + | Now for two of the testers (R and S) (See Figure 1), the web map was better than the paper questionnaire on all |
but one criterion. The exception was the organization of information on the page. | but one criterion. The exception was the organization of information on the page. | ||
| Line 21: | Line 23: | ||
''5. PBNI Equality Scheme has given a commitment that in carrying out its functions it will have due regard to the need to promote equality of opportunity between people who fall under the following groups as stated in s75 of the Northern Ireland Act 1998. To help us consider potential inequalities please identify the people in the following groups who you believe will be most affected by the proposed changes (for staff and service users):- Religious belief: Positive [ ] Negative [ ] ..'' | ''5. PBNI Equality Scheme has given a commitment that in carrying out its functions it will have due regard to the need to promote equality of opportunity between people who fall under the following groups as stated in s75 of the Northern Ireland Act 1998. To help us consider potential inequalities please identify the people in the following groups who you believe will be most affected by the proposed changes (for staff and service users):- Religious belief: Positive [ ] Negative [ ] ..'' | ||
| − | + | However, even when answering the simpler questions, the testers discussed what the questions mean before attempting to answer. | |
| − | attempting to answer. In contrast, there were few problems when using the on-line map. They found the probation office locations, recognised local features, and managed to manipulate and | + | |
| − | move around the map display. Writing in several words as a comment on a particular site was not a problem for two of them: the third got the research assistant to help him. In the discussion afterwards, they expressed their satisfaction with the map interface, and how easy it was to use. | + | In contrast, there were few problems when using the on-line map. They found the probation office locations, recognised local features, and managed to manipulate and move around the map display. Writing in several words as a comment on a particular site was not a problem for two of them: the third got the research assistant to help him. In the discussion afterwards, they expressed their satisfaction with the map interface, and how easy it was to use. |
| + | |||
Appendix 7 shows some sample comments added to locations on the map. They relate directly to the issues of the consultation, focused on specific sites. By starting from the particular, rather than requiring consultees to give general comments, it is both easier for consultees to express their views, and consulters to understand and analyse them. | Appendix 7 shows some sample comments added to locations on the map. They relate directly to the issues of the consultation, focused on specific sites. By starting from the particular, rather than requiring consultees to give general comments, it is both easier for consultees to express their views, and consulters to understand and analyse them. | ||
Revision as of 19:09, 13 January 2008
Test Method
Using the test protocol, it was possible to compare the innovative map-based e-consultation technique with the conventional paper questionnaire used in the Probation Board consultation. The comparison is shown in another chart (See Figure 2)
The chart shows the difference between the post-test scores given to the web map and the paper consultation questionnaire. The scores were 5 for strongly agree down to 1 for strongly disagree. The difference can range from +4 to -4. Where the question was a negative one, the difference was taken away from 0, so that in all cases +4 is better for the web map, and -4 is worse.
Test Results
Scores suggest that the the paper questionnaire was considered as better.
Now for two of the testers (R and S) (See Figure 1), the web map was better than the paper questionnaire on all but one criterion. The exception was the organization of information on the page.
For those with less Internet experience, the questionnaire was better. But on every other criterion of ease of use, the map-based interface was better.
The third tester (P) was over 60, and had had less experience of computers and the Internet. On some criteria he found the paper questionnaire easier (based on his experience of filling in forms in the past). Nevertheless, he was more satisfied overall with the new web map than with the traditional questionnaire.
During the tests, there were frequent complaints about the paper questionnaire. Q5 was particularly hard to grasp. It reads like a university or A-level examination question:
5. PBNI Equality Scheme has given a commitment that in carrying out its functions it will have due regard to the need to promote equality of opportunity between people who fall under the following groups as stated in s75 of the Northern Ireland Act 1998. To help us consider potential inequalities please identify the people in the following groups who you believe will be most affected by the proposed changes (for staff and service users):- Religious belief: Positive [ ] Negative [ ] ..
However, even when answering the simpler questions, the testers discussed what the questions mean before attempting to answer.
In contrast, there were few problems when using the on-line map. They found the probation office locations, recognised local features, and managed to manipulate and move around the map display. Writing in several words as a comment on a particular site was not a problem for two of them: the third got the research assistant to help him. In the discussion afterwards, they expressed their satisfaction with the map interface, and how easy it was to use.
Appendix 7 shows some sample comments added to locations on the map. They relate directly to the issues of the consultation, focused on specific sites. By starting from the particular, rather than requiring consultees to give general comments, it is both easier for consultees to express their views, and consulters to understand and analyse them.
