Usability Testing for the Non-Visual Access to the Digital Library (NoVA) Project
[ Home Page - Surveys - Documents - Presentations - Search ]
The Non-Visual Access to the Digital Library (NoVA) project was concerned with countering exclusion from access to information, which can all too easily occur when individuals do not have so-called 'normal' vision. Usability tests undertaken for the NoVA project provided an insight to the types of problems faced by users and interestingly, although the focus of the project was on the information seeking behaviour of blind and visually impaired people (generally using assistive technologies), the control group of sighted users also highlighted usability problems. This showed that although awareness of Web accessibility is increasing, all types of user could be faced with navigational problems, thus reinforcing the importance of involving a variety of different users in any design and development project.
Some problems experienced were due to accessibility and usability conflicts such as inappropriate or unhelpful use of alternative text, or poor use of language. Other problems were due to a lack of understanding of the different ways users interact and navigate around Web-based resources. Careful consideration must therefore be given not only to assure conflicts between accessibility and usability are addressed, but to the layout and navigation of a site and to the ways different assistive technologies interact with them.
This case study will look specifically at the usability testing phase of the NoVA project. The final report of the NoVA project [1] fully describes the methodology, findings and conclusions, and outlines a number of recommendations for digital library system design.
Despite evidence of much good work to make interfaces accessible and on methods for accessibility checking (see for example: EDNER, 2002 [2] and the World Wide Web Consortium Accessibility Guidelines [3]), there is less work published on usability issues or how people using assistive technologies (such as screen readers) navigate around the Web interface.
Although sites may adhere to accessibility recommendations, users can still experience navigational problems. This is partly due to the fact that Web pages are increasingly designed for parallel or non-serial navigation, offering a variety of options within one page (frames, tables, drop down menus etc). Parallel design can cause problems for users who are navigating the site using assistive technologies which force them down a serial (or linear) route, for example a screen reader reading out every hypertext link on a page one by one.
The overall objective of the NoVA project was to develop understanding of serial searching in non-serial digital library environments, with particular reference to retrieval of information by blind and visually impaired people. Serial searching was defined for the project a linear movement between pages, non-serial (or parallel) searching was defined as movements around a page, between frames or interacting with a number of options such as a table, dialog box or drop-down menu.
Using a combination of desk research, task analysis and user interviews, the objectives of the study were to:
The NoVA usability tests used a combination of observation, transaction logging and verbal protocol, together with pre-and post-task questions.
A sample of 20 sighted and 20 blind and visually impaired people was used to undertake a series of usability experiments. Definitions of terms were set at the beginning of the project. The 'sighted' sample was made up of users who were all able to read a standard (14" - 15") screen. The term 'visually impaired' was defined for the NoVA project as people who needed to use assistive technology, or had to be very close to the screen to be able to 'read' it. The sample size for the NoVA project enabled comparative analysis to take place between two user groups, however it should be noted that Nielsen (2000) [4] suggests excellent results can be obtained from usability tests comprising as little as five users (although he recommends at least 15 users to discover all the usability design problems).
Four information-seeking tasks were set using four Web-based resources:
Although not all of these might be viewed strictly as digital library resources, each resource displayed elements of parallelism in their design and were generally accessible, to greater and lesser degrees, according to the WAI recommendations.
Each of the tasks was consistently set so that comparative analysis could take place between the sighted and visually impaired users. For example, users were asked to search for a national and regional weather forecast using the same search engine.
It was recognised that success in performing searches could be influenced by previous knowledge or experience, either of the technology, the site visited, the subject matter of the task, or by individual interpretation and approach to a task. In an attempt to obtain a balanced picture, the tasks set covered a fairly broad subject base such as weather forecasts, shopping for clothes and travel information.
Every attempt was made to create a relaxed atmosphere and to dispel feelings among the users that they were being tested in any way (although inevitably this still occurred to some extent). This included an initial explanation of the purpose of the study, i.e. to highlight Web usability issues rather than to test information seeking skills. The observer also chatted informally prior to the tasks and offered the users tea/coffee and biscuits to put them at ease. Naturally, the users were ensured that all their responses would be kept strictly anonymous and only used for the stated purpose of the study.
To ensure everyone started from the same place, users were required to commence using the stated electronic resource, but were allowed to choose whether they used the search facility or browsed the site for relevant links. So for example, when asked to look for the national weather forecast for the UK, users were required to start with the search engine, either by typing in search terms or by browsing for a relevant weather link.
Users were not given a time limit to complete each task. At the beginning of the session they were told that they could stop the task at any time and were given examples such as "if you are satisfied that you have found the information", "if you are not satisfied, but think you have found all the information there is", or "if you are fed up with the task". The reason for this was to try and simulate real-life information searching behaviour, where information required by a user may or may not be found from within a specific resource and was not a judgment of the amount of information retrieved.
Data was gathered using a combination of on-screen data capture (Lotus ScreenCam which records on-screen activity and verbal dialog), sound recording and note taking. This method enabled each task to be recorded (either on-screen or by the sound of the assistive technology with verbal dialog) and backed up by note taking.
Users were asked to verbally describe what they were doing during each task. Users were also asked a set of pre- and post-task questions. These comprised general questions, such as how to tell a page is loading, initial comments about the interfaces and the type of information provided; and usability questions, such as their overall experience navigating around the resource. Both the verbal protocol and the pre- and post task questions provided a richer picture of the user's experience by enabling the observer to ascertain not only what they had done, but why they had done it, and how they felt about it.
Interviews were conducted before and after each task to help ensure the electronic resource and the task performed were still fresh in the user's mind before moving on to the next resource.
Data was transcribed in two ways:
Data from the searches and questions were entered and coded into a Computer Assisted Qualitative Data Analysis tool (Atlas-ti) [5].
Data was analysed using Atlas-ti analysis tool, which provided an efficient method of data storage and retrieval. Although entering and coding data was initially time consuming, once completed it provided quick and easy access to the large amounts of data gathered for the project. It was then possible to generate queries and reports for data analysis and report writing.
Each step taken during a search was coded to show the number and type of keystroke used within each search task. This was used to compare the information seeking behaviour of the two samples (sighted and visually impaired) and to look at different trends within each.
Data from the pre- and post-task questions was grouped and coded into categories. This enabled comparisons to be made relating to specific questions. For example, coding quotes from users relating to the question 'How do you know if a page is loading?' revealed that only one of the sighted users mentioned the listening to the hard drive, whereas many of the visually impaired users said they relied on this clue to tell them that the page is loading.
Gathering volunteers for the study was a time-consuming process and could have been a problem if it had not been built in to the NoVA project time frame. It is therefore worth bearing in mind that a substantial amount of time and effort is needed to gather a suitable sample.
In order to obtain specific data on the way people search electronic sources, it was necessary to select a sample of people who were reasonably familiar with using the Internet and, where appropriate, were comfortable using assistive technology. This meant that it was not possible to gather a true random sample. Although this was not particularly problematic for the study, it did mean that the results could not be generalised to the population as a whole.
Data was gathered using a combination of on-screen data capture (Lotus ScreenCam [6]), sound recording and note taking. Initially it was hoped that ScreenCam could be used throughout, however the pilot tests revealed that ScreenCam can interfere with assistive technologies, so it was necessary to use a combination of sound recording and note taking for the visually impaired sample.
It was difficult to create a natural environment for the users to perform the tasks, and although every attempt was made to make the users feel comfortable and to dispel any feelings that their ability was being tested, inevitably at times this did occur. However, this problem was probably unavoidable for the capture of qualitative data.
The observer attempted not to prompt subjects or give any instructions while the subject was performing the task. This proved difficult at times, particularly when it was evident that the user was becoming distressed. In some cases the observer had to provide a "hint" to enable the user to continue (it is suggested that this type of intervention is sometimes necessary in certain circumstances, as is prompting a user to ensure the transcription is accurately logged [7]).
The usability tests undertaken for the NoVA project provided a rich picture of the types of problems faced by users when navigating around Web-based resources, particularly when using assistive technologies. It also provided evidence of the types of features users liked and disliked, how they overcame navigational problems and what types of features enhanced their searching experience, all of which can be fed back into recommendations for the design of digital library systems.
Although the sample chosen was appropriate for the NoVA study, for general usability studies it would be desirable to try to include users with a variety of disabilities such as motor impairments, hearing impairments and visual impairments. Also, users with a mix of abilities to ensure the site is usable as well as interesting and engaging. Usability testing should be used alongside accessibility checking to provide a rich picture of the accessibility and usability of a Web site, which will help designers and developers to ensure their sites embrace universal access and access for all.
The findings of the NoVA usability testing, together with conclusions and recommendations are described in the final report, which is available to download from the NoVA project Web site in Word, PDF and HTML [1]
Jenny Craven
Research Associate
CERLIM, Manchester Metropolitan University
Tel: 0161 247 6142
Email: j.craven@mmu.ac.uk
URL: http://www.mmu.ac.uk/h-ss/dic/people/jcrhome.html
[ Home Page - Surveys - Documents - Presentations - Search ]
Web page by Brian Kelly of UKOLN.
Last Modified 21-June-2004
Email comments to webmaster@ukoln.ac.uk
QA Focus Comments
For QA Focus use.