Appendix VII: User surveys

For each service a description of the survey aims, methods and response is given, followed by a summary of the findings, if appropriate, in three critical areas:

• User requirements
• User behaviour
• User satisfaction

A number of points about these user surveys should be borne in mind:

• The response rates vary from over 200 to less than 30. The level of response should be noted carefully when considering the findings.
• The aims of the surveys vary: some were designed to feed into the development of the service; some were done as part of the evaluation process; and some were not directly associated with the service but to a related project.
• The different aims of the surveys means that some surveys did not directly cover the user-factors related to the development of resource selection criteria.
• The subject gateways vary considerably in their stages of development.

1. ADAM

In the early part of 1996 ADAM conducted a questionnaire survey of users and potential users via the ADAM server and an email shot of a number of discussion lists. Results are due in Autumn 1996 (to be examined in a later stage of DESIRE).

2. EELS

User behaviour

Some statistics taken from the period between the 12th of March 1995 and the 18th of June 1995 show that the majority of users browse the system for resources as opposed to searching (32,000 accesses browsing, 3,500 searching). Although it is suggested that probably the majority of the browsing accesses are in fact users finding pages on the system from outside the service itself i.e. they are coming from WWW search engines. The main usage by domain/country is from Sweden (between 50-75%)

3. EEVL

In April/May 1996 EEVL surveyed a selected sample of users from 6 pilot sites (UK universities). The survey was conducted via workshops held at the sites, with three stages to the data collection:

• A background questionnaire (before having used EEVL)
• A group discussion
• An initial impressions questionnaire (after having used EEVL)

The aim of the workshops was to introduce the EEVL system , and to carry out evaluation of the pilot phase of the project. Responses were collected from 81 users.

User requirements

The survey elicited a list of the types of information that users were searching for on the Internet:

1. Users wanted to find information on other people/departments doing similar research
2. Full text of reports/papers
3. Company information -particularly information on products
4. Software (FTP sites for downloadable software)
5. Engineering education
6. Conferences
7. Search tools like BIDS
8. On-line journals
9. FAQs
10. IEEE abstracts
11. Technical data i.e. electronics datasheets

12. Physical property data

- In general participants were supportive of not including newsgroups in the database.

User behaviour

a) Behaviour when using the Internet generally:

• The majority of users said they used computers every day.
• Respondents felt they were experienced users of email and word-processors, but less experienced users of FTP and gophers. Most said they were of 'average' experience at on-line searching.
• Despite wanting to find information on the Internet, users often refrained from searching because they felt 'the costs of locating the information outweigh the usefulness of the information itself'. Users found searching slow because of the slow connections and the unfocused nature of the 'hits' from search engines.

b) Behaviour when using EEVL:

• Users found the searching and browsing facilities easy to use.
• They tended to use the search facility more than browse, although they used the browsing to get a feel for the range and depth of material in the database.

User satisfaction

• The initial reaction to the EEVL service was 'overwhelmingly positive'.
• Users commented that they liked the inherent simplicity of the system.
• They expressed a desire to access more information via the Internet and recognised that EEVL could save them time in doing so.
• A service which focused on quality resources in engineering was welcomed.
• Two concerns were raised over the size of the database (small), and the future of the service after the end of the two year project.

4. OMNI

An on-line user survey was conducted as part of the ROADS evaluation process in June 1996. The aim was to gauge the users of the ROADS system. A questionnaire was placed on the OMNI server for a limited period. There were 23 useable returns, with 16 UK universities represented.

User requirements

Users were asked what other services they would like to see on OMNI and what type of resources they would like to find. The answers included:

• 'Ability to choose purely British sources'
• 'More of the same, but in different subject areas'
• 'increased coverage' (generally and in specific subject areas)
• Respondents worked in a wide range of different disciplines suggesting a breadth of coverage is needed.

User satisfaction

Things that users thought were particularly good about OMNI:

• 'the fact that sources had been evaluated, and the resulting quality information '(the quality control)
• the range and types of resources available
• the well organised structure

Things that users thought were not particularly good about OMNI:

• the small number of resources available on the service

In general users' comments suggested that users are pleased with what the service is providing and wish for more of the same.

5. RUDI

A paper questionnaire was sent out in February to all UK Higher Education institutions (62) which are known to have teaching and/or involvement in one or more areas of urban design, as well as an enclosure in Urban Design Quarterly to reach professional bodies and practitioners. (The main aims of the survey were to:

• define the requirements of RUDI by end-users
• provide a basis to develop a strategy for selecting and organising material on RUDI

This report is based on the provisional results of the survey.

User requirements

RUDI differs somewhat from most of the subject gateways in their intention to host primary material on their server, in addition to a gateway to resources available elsewhere. Many of the questions were to establish the type of material users would like to see on the service.

• There was a marked preference for UK based information although material on Europe and North America was also required.
• Most respondents would like to have the facility of an electronic noticeboard.
• Respondents were asked to specify the likely end use of material on RUDI, most academics indicated teaching, closely followed by research. Commercial practitioners however would use it to develop their own design ideas.

User behaviour

The project is still in its preliminary stages and only publicity material is available at this stage. However with regard to usage, results from the preliminary survey suggest that current usage of the Internet amongst design schools is relatively high and anticipated use of the RUDI service is also high. Almost all of the academic respondents said they were likely to use the system as well as a majority of the practitioners.

6. SOSIG

There have been two user surveys which relate to SOSIG:

• Evaluation forms/anecdotal evidence collected via Internet workshops prior to the establishment of SOSIG (1992-4).

• An on-line survey done as part of the evaluation of the ROADS project (1996).

The survey preceding SOSIG

Over 200 evaluation forms and a wealth of anecdotal evidence were collected from users by Nicky Ferguson when he conducted Internet workshops in UK universities between 1992 and 1994, prior to the SOSIG project. Working for the ESRC, Ferguson conducted workshops entitled 'The Internet for Social Scientists' and 'Exploring the Networks and Accessing Information'. His observations of users and the feedback he obtained from them are summarised below.

User requirements

• Users want a simple way of finding Internet resources that can help them with their work.
• Users want to be able to find resources of a high quality without having to spend lots of time browsing the Internet.
• Users want a 'one-stop-shop' for social science resources so that they can locate resources of a high quality, quickly and easily.
• Internet resources need to be accessible to users with any kind of network connection and any kind of hardware.

User behaviour

• Anecdotal evidence from Nicky Ferguson suggests that users tended to find it easy to browse the Internet but difficult to find anything of actual use to their work. Browsing for useful resources was not proving to be an effective use of their time.
• The workshop evaluation sheets suggest that some users did not feel confident that they were finding the Internet resources most appropriate to their work. They lacked awareness of the range and number of resources available to them via the Internet and lacked the confidence to use the Internet in the absence of the trainer.

User satisfaction

This survey preceded SOSIG so user satisfaction could not be measured.

The 1996 survey of SOSIG users.

An on-line questionnaire was mounted on the SOSIG server for two short periods in 1996 (For 3 days in January and for a week in July) as part of the ROADS evaluation process. The aim was to gauge the users of the ROADS system. Users were invited to fill in the questionnaire on a voluntary basis, and there were 40 useable returns.

User requirements

Users were asked what other services they would like to see on SOSIG and what type of resources they would like to find. The answers included:

• 'more of the same high quality material.'
• 'indexes kept in frames while browsing/searching'
• 'more regular updates'
• A number of requests were made for more resources to be available in particular subject areas
• In terms of the types of resources that users wanted to find on SOSIG, the survey found that users wanted 'more of the same'. Users want more of the same high quality material.
• Suggestions for improvements included: 'more international resources'; 'a social science software site'

The respondents were from a wide range of disciplines within the social sciences, suggesting a breadth of coverage is required.#

User behaviour

• Most of the users used SOSIG for their own use, but 10 users were carrying out searches for other people.
• Most users were using SOSIG for research purposes (26/40), as opposed to teaching or learning purposes.
• Generally users liked having both browse and search facilities, and found them both easy to use.

User satisfaction

Things that people thought were particularly good about SOSIG:

• 'quality of resources'
• 'discipline based structure'
• 'ease of use and evaluation of sources'
• 'links to other useful sites'
• 'URL descriptions'
• 'clear structure'
• 'range of coverage'
• 'tree structure'
• 'appearance'

In terms of the quality of the information 31/40 users felt the resources found on SOSIG were either 'good' or 'very good'. No users found the information to be of a 'poor' quality.

Things that people thought were not particularly good about SOSIG:

• 'Not many links yet'
• 'not updated enough'
• 'sometimes difficult to find certain links listed'
• 'need to turn back pages to find resources on the same index'
• 'finding by author's name'
• 'no indication of the number of hits from a search'
• 'poor description of what the SOSIG database offers'

Conclusions

As noted in the introduction most of the surveys were not designed to elicit feedback related to resource selection criteria (although OMNI have work in progress with Alison Cooke of Aberystwyth University on end-user perceptions of information quality in the networked environment). However, some of the existing results may help indirectly. For example, a number of the surveys found that users were highly satisfied with the quality of the services as they stood (e.g. OMNI, EEVL, SOSIG), suggesting that users were satisfied with the selection criteria being used by those services. Similarly, some surveys yielded information on the nature of the user population, which could imply that certain types and qualities of resources would be required to meet certain needs. The EEVL survey produced a list of the types of information that users were searching for on the Internet providing a focus for resource selection.

Next	Table of Contents

Page maintained by: UKOLN Metadata Group,
Last updated: 2-Apr-1998