Evaluation of the Electronic Libraries Programme
Synthesis of Annual Reports

• 2. ACTIVITIES AND PROGRESS
  • 2.1 Generic Activities
    • 2.1.1 Staffing
    • 2.1.2 Management structures and processes
    • 2.1.3 Technical and legal infrastructure
    • 2.1.4 Project planning and review
    • 2.1.5 Promoting the project
    • 2.1.6 Continuous formative evaluation
  • 2.2 Domain Specific Activities
    • 2.2.1 Electronic journals
    • 2.2.2 Access to network resources
    • 2.2.3 On-Demand publishing
    • 2.2.4 Digitisation
    • 2.2.5 Electronic Document Delivery
    • 2.2.6 Training and Awareness

2. ACTIVITIES AND PROGRESS

In providing an overview of the progress of eLib projects during the first year of the programme, we focus primarily on the generic activities in which they have all been involved and which to a large extent characterise the start-up phase of any innovation project. Within domain clusters, projects were also engaged in similar activities and we briefly outline these in Section 2.2, Domain Specific Activities, below.

2.1 Generic Activities

For the large majority of projects, the period covered by the first annual report was primarily concerned with getting the project up and running. For a small minority, eLib funding supported an already established or embryonic activity and so these projects were able to move more rapidly towards implementation. Nearly all the first round projects had progressed beyond the start-up phase to implementation and the achievement of project milestones, although a small minority experienced continuing problems with staffing and technical equipment which delayed their progress.

The activities during this start-up phase can be grouped as follows:

• • staffing
• • establishing management structures and processes
• • developing technical and institutional (legal) infrastructure
• • project planning and review
• • promoting the project
• • continuous formative evaluation

2.1.1 Staffing

Appointing new staff and assembling a project team was a primary activity for all projects, readily accomplished by some but a source of ongoing difficulty for others due to difficulties of recruitment and staff turnover. This reflects the scarcity of the particular skill profile required by eLib projects and consequent high marketability and labour market mobility of such persons.

The composition of project teams or staff groups varies widely as does their mode of functioning. The models of staffing adopted by projects include varying combinations of the following personnel:

• • dedicated project staff (full-time or part-time) on contract for the duration of the project or on secondment
• • dedicated project staff (part-time) who are engaged to work on eLib and other linked IT projects
• • staff in permanent positions in the partner organisations (primarily in universities and professional associations) with a significant project role and whose involvement is remunerated in some way
• - advisers and consultants external to the project partners whose contribution is remunerated
• - persons making an unpaid or uncosted contribution to the project, whether as project directors, members of editorial boards, members of user group panels, trouble-shooting computing staff etc.

Although most projects have a small number of paid dedicated project staff (most commonly between three and five), the project team size is augmented by these other categories of personnel who in many cases greatly outnumber staff appointed to project posts. Some project teams have as many as ten to thirteen members, distributed across different partner institutions and organisations.

A common staffing pattern includes the following role-holders and areas of expertise:

• • project director(s)
• • project manager or coordinator
• • one or more staff with expertise relevant to the domain area (editorial production, training, research, copyright, bibliographic, etc.)
• • a technical officer and/or programmer
• • clerical support

Depending on the area, the project team might be supported by teams or individuals in partner institutions developing courseware materials, delivering training, undertaking editorial work, providing an operational service. Much of this work is undertaken on a voluntary basis, underlining the success of eLib in leveraging additional resources.

Several projects have already reviewed their staffing profile, in some cases deferring appointments to a later stage in the project life-cycle, in others re-combining roles and in yet others identifying the need for quite different kinds of skill. Evaluation is an area where several projects feel they lack appropriate expertise and will need to redirect resources to this area.

The annual reports draw attention to the quality, professionalism and enthusiasm of the personnel involved in making their project a success, whilst alerting eLib managers at the same time to the substantial unpaid time input which is unlikely to be sustained in the longer term and to the evident under-staffing in some projects. The latter tends to be compensated for by over-commitment of those working on the project, already leading in some cases to burn-out and consequent staff turnover.

2.1.2 Management structures and processes

Virtually all projects have set up a Steering Committee of some kind (variously named Project, Advisory or Management Board, Project Advisory Group, Executive Committee, Steering Group) which meets at regular intervals, usually quarterly or half-yearly.

As one might expect from such diverse projects, the actual composition and size of Steering Committees varies. Not all annual reports included information about membership, however, so our overview is partial. In several projects, the membership is exclusively drawn from academic and library staff in universities who are usually representative in some way of the intended user base; others have additional members from relevant professional associations and key organisations such as the British Library; a smaller number have a more balanced membership that includes publishers and other commercial organisations. As required by the conditions of funding, project steering committees include FIGIT representation.

There is relatively little information in the annual reports about the functioning of steering committees. Our impression is that projects primarily look to their steering committee for advice on policy and overall strategy although in some instances the members are involved in a hands-on way.

The Tavistock Institute’s Evaluation Guidelines for eLib Projects emphasises the importance of a Steering Committee with membership and function strategically chosen to enhance uptake and exploitation of project outputs and learning. Some projects have clearly been strategic in their choice of members whilst in others it would appear that membership is based more narrowly on familiar networks. One conspicuous input missing in many projects is that from university senior management. Occasional presentations to senior management are likely to be important for the longer term integration of eLib innovation beyond the confines of the library and participating individuals and departments.

The most common arrangement for project management is a project team, overseen by a project director and with operational responsibility vested in a project manager or coordinator. Variations on this arrangement include a project executive group supported by standing ‘working parties’ or ‘project assurance teams’ with different areas of responsibility. In the Electronic Journals area, an Editorial Board often sits alongside the project management group.

Project management is complicated by the dispersal of staff across different institutional sites, and by the varying combination of dedicated, part time, and voluntary or unremunerated staff. At a purely pragmatic level, finding a suitable date for a meeting can itself be quite problematic. One or two reports commented on the time it takes to build effective modes of communication and decision-making and to establish protocols and routines that underpin smooth management. Most projects combine periodic face-to-face meetings of the project team with one or more modes of electronic communication, including occasional use of tele- or video- conferencing. In a number of cases, projects had hoped to use e-mail as a substitute for meetings but found this arrangement unsatisfactory.

2.1.3 Technical and legal infrastructure

Installing and customising technical hardware and software has been a pre- occupation of most projects, particularly those dealing with applications that use advanced rather than stable technologies or which call for complex systems integration. Many projects have experienced difficulties in establishing the basic technical infrastructure, commonly as a result of delays in delivery of equipment, missing parts or faulty software. Further set-backs have arisen from projects’ dependency on wider institutional technical infrastructure, including technical advice and support services, which is often inadequate for the functionalities being developed.

Specification of the systems architecture, including investigation of technical choices and the production of technical documentation, has been an early start-up task. Project staff have also had to familiarise themselves with new hardware and software and learn the new skills associated with designing Web pages, the production of PDF files, the marking up of HTML, etc.

On the legal side, several projects have developed model agreements or memoranda of understanding between the partner organisations and in one case the project has established itself as a legal entity. Others, however, have preferred to function with more informal arrangements between partners. Additional aspects of legal infrastructure include making provision for indemnity against copyright infringement by authors as well as the installation of security systems.

2.1.4 Project planning and review

The start-up activities of projects have often included the development of a work plan, and the elaboration of an evaluation plan that takes account of the Guidelines on Project Evaluation and the evaluation workshops and clinics. Far fewer projects have developed a business plan for the downstream uptake and exploitation of products and services, although projects have been encouraged to plan for sustainability at the outset rather than as an end-phase activity.

A small minority of projects brought the partners together early on to review the project concept, to clarify how far they have a shared understanding of the innovation and to make explicit what partner organisations hope to gain from their involvement in the project. Where this has occurred it has been seen as a fruitful activity, although there are other projects which clearly would have benefited from this degree of collective reflexivity in the early stages of the project. Several annual reports have made reference to difficulties in communication between the partners that reflect different expectations or understandings as well as different modes of discourse.

2.1.5 Promoting the project

Awareness-raising has been a significant area of activity for all projects. In general, promotional activities are focused on some tangible product or service that is the intended outcome of the project, and are aimed at creating a receptive climate for their innovation and at enhancing the prospects for user acceptance. The broad strategic actions adopted by projects as part of general awareness raising and project promotion are generally multi-pronged, including some combination of the following:

• - publicity
• - training of potential users
• - networking among influential opinion leaders and innovators
• - rapid prototyping

Publicity

All projects have developed promotional material and engaged in a variety of activities aimed at raising the profile of their project. Profiling has generally included the adoption of logos and creation of a distinctive identity but has also extended to merchandising. The most common activities reported by projects in descending order are:

• - design of Web pages
• - press releases and articles in national newspapers
• - project presentations at conferences, eLib meetings, etc.
• - journal articles
• - distribution of brochures and flyers on the project
• - participation in conferences

Projects have also been active in setting up electronic communication tools as an awareness raising device including Web advice point, newsgroup archive, e-mail discussion list and closed list on Mailbase.

Training

Training events have also been used by projects as a strategy for awareness raising. Participants at these events are introduced to Web and Internet use and to the benefits that the project innovation has to offer in an electronic environment.

Networking

Projects make use of existing networks as a means of promoting the project and raising awareness about developments in electronic libraries more generally. Many projects adopt a cascade model of networking, focusing on the gatekeepers, people with status and credibility within the professional or academic discipline and those in pivotal positions, in the expectation that with their active support and endorsement the project will more readily gain user acceptance and diffuse more widely.

2.1.6 Continuous formative evaluation

Evaluation, in the sense that we use it here, is integrated into a user-centred development process in which feedback from the user population informs the design and ongoing development of the software application in an iterative way.

Key features of a user-centred approach are:

	1. knowledge of the real users or the potential user population and the context in which the innovation is being introduced
	2. feedback from users at key stages in the development lifecycle that informs the ongoing development of the innovation
	3. involvement of key stakeholder groups, including the various user communities, in decision-making about design choices
	4. strategies for empowering users so that they can exercise a genuine user influence over the systems development process

Whilst ‘best practice’ in systems development is moving towards a coherent user-centred design process incorporating all the above, more generally users are seen as sources of feedback rather than active participants. The annual reports present a similar picture for eLib systems development practice: there are some notable examples of projects where the end users have a real stake in the systems being developed and are empowered in the dialogue with developers, but the more general picture is one of involving users primarily as sources of feedback. It is important to acknowledge this widespread commitment to iterative feedback processes as exemplars of good practice, even if not yet matching up to best practice in the eLib programme.

Only a few projects made any mention of a systematic profiling of their intended users or an analysis of the context in which their product or service is to be used, whether of the intended future user population or more narrowly of their pilot sites. In those cases where data on the user population was gathered, the methods used were fairly simple ones:

• • a survey of a random sample of members (of a professional association) on their level of Internet use
• • use of training sessions to gather information about users on an informal basis
• • an on-line questionnaire inviting users to profile themselves
• • a survey of user information needs

In relation to pilot sites, three projects referred to the nature of their information gathering activities, these included:

• • analysis of site and user context together with a profile of the user test population
• • a survey to establish the usage of the current electronic library facilities and user reaction to an electronic library environment
• • a preliminary visit to test sites to determine the quality of facilities

Many more projects however have gathered feedback from users on such aspects as the user interface, user requirements, reasons for non-use or take-up of the facility, and areas for improvement. The methods are quite diverse, some of them being one-off surveys at a fixed point in time, others of them gather data on an ongoing basis, and yet others entailing a distinctive structured arrangement for associating users with the development process.

The OMNI Approach to User Centred Development The OMNI projects exemplifies a user-centred approach, formalised through a Senior User, a Project Assurance Team, Advisory Groups, Volunteers and Focus Groups: "Right from the start, it was felt that user involvement needed to be established throughout the development of the service. This is especially highlighted by the inclusion of a user representative (the Senior User) in the OMNI Project Board and Executive Board, who has advised on all aspects of OMNI development throughout the life of the project. Additionally, a Project Assurance Team (PAT) was convened, which includes representatives from two key user constituencies for OMNI: the University Medical School Librarians Group and the University Health Sciences Librarians, as well as the Project Researcher from SOSIG. Subsequently, the PAT has been broadened to include an additional volunteer member from the user community and also a nominee from the Centre for Information Quality Management. The remit of PAT has been to supply independent preview and review of OMNI user-oriented products, primarily the user interface, and resource descriptions, user documentation, and presentations and workshops. Two key areas were additionally identified in which it was felt that OMNI could benefit from direct contributions: the development of evaluation criteria by which OMNI staff and volunteers could assess the value of network resources for inclusion in the OMNI database; and issues relating to indexing and classification of resources selected for inclusion. Consequently, two Advisory Groups were convened to support the work of the OMNI team in these areas (email discussion lists were specially created). The volunteer effort has been a cornerstone of OMNI activity, primarily aimed at information professionals, although practitioners have also participated. Two Volunteers’ Workshops have been held, designed to address volunteers’ special training and awareness needs. The OMNI-collaborators Mailbase list supports OMNI contributors and offers a useful platform for the discussion of issues that arise. Finally, in order to listen to users directly, focus groups have been planned with three user constituencies."

The methods we came across in the annual reports included:

• • questionnaires, interviews, observations of use, think aloud sessions and focus group discussions, aided in some instances by hands-on experimentation with a demo or prototype
• • a longitudinal study of users that follows users back into their working environments to assess likely uptake and exploitation
• • expert ‘connoisseurship’ whereby the leading scholars in the field provide feedback on different facets of the application
• • the gathering of informal comments made by users during training sessions or in the course of relatively unstructured demos at conferences
• • creation of an on-line mail list and discussion group whereby users are invited to contribute their ideas, comments, reactions to what is being proposed or piloted
• • on-line questionnaires that record usage patterns as well as user reactions
• • the recruitment of a test panel for a disciplinary area to provide systematic, structured feedback
• • a Volunteer Programme
• • the recruitment of Interested Persons (TIPs)

It is not clear from many annual reports how the data gathered are fed back into the development process, or whether users have any input into the key design choices as distinct from general usability aspects. A small number of projects are however explicit about these links, reflecting their mature user-centred design approach. In two cases, the elected chair of the User Group is represented on the project Steering Committee; in another, the user survey working group is one of four working groups that feed directly into the project executive and steering committee.

2.2 Domain Specific Activities

2.2.1 Electronic journals

The specific activities undertaken by projects in the EJ domain are clustered in ways that reflect the different purposes of EJ innovations. Those projects that seek to establish an electronic journal in a disciplinary area have been engaged in a similar set of activities which include:

• • attracting subscribers through a variety of awareness-raising actions
• • promoting awareness of the journal within the discipline and the publishing world
• • forming an editorial board and establishing the procedures to support the editorial process
• • drawing up an agreement to cover liability for work published
• • generating material for the journal
• • engaging in debates about such issues as subscriptions, management of copyright and intellectual property rights, refereeing etc.
• • putting feedback mechanisms in place

2.2.2 Access to network resources

Projects in this domain are engaged in a similar set of activities, although the established nature of networked resources in some disciplinary areas has allowed some projects to focus more on refining the interface and facilitating the creation of new networked resources than on developing the services specification and technical capabilities of the service. One project, dealing primarily with grey material in a politically sensitive, has found considerable copyright problems. The common ANR activities of projects include:

• • undertaking user needs requirements resource discovery and cataloguing
• • development of resource descriptions
• • provision of a test bed for ROADS as part of piloting of a prototype
• • development of training materials and conduct of training
• • facilitating the creation of new networked resources
• • the production of technical documentation

The particular successes reported by projects reflect their individual priorities and identities. In broad terms, successes in this first year relate to:

• • achievement of targets within or ahead of scheduled timeframe
• • positive user feedback on interfaces

• technical development, including for example the shift from a general purpose WWW server software to a custom-server optimised for delivery of template database

Access to Network Resources Demos, prototypes & trials [ Not necessarily a comprehensive listing.] Piloting of the Edinburgh Engineering Virtual Library in six test sites Launch of the OMNI medical database with over 700 resource descriptions, and seminar for 170 delegates Demonstration of the ROADS database software in SOSIG, the social science gateway, with over a 1,000 quality resource descriptions Launch of ADAM’s prize winning descriptive Web site Some notable achievements ‘1,000 UK accesses per week achieved within three months’ (IHR-Info) ‘Impressive response to training workshops and materials in use of Internet for social scientists’ (SOSIG) ‘Early provision of software for stable and well designed subject gateways.....Major participation in, and stimulation of, standard making activities’ (ROADS)

2.2.3 On-Demand publishing

The ODP projects span a range from the more technical, experimental applications to those primarily concerned with the implementation in real environments of ODP courseware based on low cost stable technologies. Although most projects are engaging in a similar set of activities, the prioritisation among them reflects their positioning on the experimental-implementation spectrum. Common activities are:

• • technical and design work for document management systems
• • development of copyright tracking systems
• • establishing relations with publishers and developing model agreements for payment systems
• • monitoring usage patterns, linked with pricing models
• • integrating ODP applications into existing technical systems

The achievements reported by this group of projects relate as much to ‘understandings’ gained as to successful outputs or outcomes. Among those mentioned are:

• • production of courseware packs with up to 60 rights holders and copyright clearances
• • successful Roadshow on ODP courseware sponsored by a commercial partner
• • comprehensive research report on copyright control for electronic course materials, benefited by collaboration with the OU

• evidence of a rethinking of strategies towards course pack provision by suppliers of document management systems

On Demand Publishing Demos, prototypes & trials [ Not necessarily a comprehensive listing.] Open Learning Foundation materials in two subject areas mounted on eOn server Creation and field testing of Web mounted electronic course booklets for three Spring semester modules on Media, Critical and Creative Arts Production of five SCOPE course packs for undergraduate sociology students at two Scottish universities, involving negotiation with nearly 60 rights holders Some notable achievements ‘Research undertaken with suppliers of document management systems resulted in suppliers rethinking their own strategies towards course pack provisions’ (Phoenix) ‘Development and implementation of a ‘four stage’ security system, while not making illegal use impossible, is nonetheless recognised by publishers as a reasonable compromise between ‘free for all’ and technical restrictions which undermine the advantages of electronic materials’ (ODP in the Humanities)

2.2.4 Digitisation

The two digitisation projects included within this overview were dealing with different kinds of materials that raised different copyright issues. Even so, there was a similarity in the activities undertaken during the first year:

• • installation and configuration of servers
• • early trials in scanning materials and making them available on Web servers
• • contacts with publishers over access to materials
• • exploration of copyright issues in particular fields

2.2.5 Electronic Document Delivery

The projects in this domain vary considerably in their stage of development, one or two being well advanced with field trials whilst others are still developing the technical configuration. At least one project is still seeking clarification of its overall concept and logic. The activities of the project in the first group includes:

• • conduct of field trials in realistic learning (and business) environments
• • development of a database and detailed recording of all transactions
• • regular feedback to participating libraries on usage patterns and comparison with BLDSC

The projects which have not yet arrived at an implementation phase are involved in such activities as:

• • planning and specifying the format of the catalogue, server content and mechanisms for transferring material
• • design of a tariff file to provide subscription and pricing information
• • consideration of different pricing models
• • development and installation of software
• • putting the evaluation in place

None of the EDD projects reported any ‘noteworthy achievements’ as yet in their annual reports, though where field trials have taken place this may be seen as an achievement in itself.

Electronic Document Delivery Demos, prototypes & trials [ Not necessarily a comprehensive listing.] Eurotext now contains 27 titles on the topic of Environmental Policy Trial of the prototype LAMDA service in nine participating institutions, followed by an extension to a wider group of 15 request only libraries

2.2.6 Training and Awareness

The T&A projects adopt a wide range of strategies for changing the attitudes and practices of librarians and other actors in the higher education system. Those employing a training strategy have prioritised training needs analysis or information gathering followed by the development and piloting of course materials and workshop events as key activities. Other activities are specific to the individual projects and include:

• • negotiations re accreditation of the course
• • literature reviews
• • network building
• • development of quality assurance manuals and procedures
• • survey of training, labour market and occupational futures
• • establishment of editorial and production systems

Table of Contents

The Electronic Libraries Programme (eLib) was funded by the Joint Information Systems Committee (JISC)
Page version: 1;
Web page maintained by UKOLN Systems Team and hosted by UKOLN - feedback to systems@ukoln.ac.uk .
Site last revised on: Tuesday, 24-Nov-1998 14:21:04 UTC
DC Metadata

Last updated 04-Apr-1997