ELECTRONIC

LIBRARIES

PROGRAMME

SYNTHESIS OF

1998 PROJECT

ANNUAL REPORTS

FINAL

TABLE OF CONTENTS

Summary

Introduction

The syntheses of the annual reports of the eLib projects are intended to summarise and communicate programme learning as an aid to programme steering and for dissemination to a wider audience.

In this, our third synthesis of annual reports, it is possible to make some judgements as to what extent eLib has achieved its programme aims: In our view eLib has provided the basis for the organisation and delivery of library systems based on electronic systems required by Follett and has changed how actors perceive the current situation and imagine the future.

On demand publishing, electronic short loan and electronic document delivery

The ODP and ESL project learning highlights many of the social, cultural and institutional issues which participating institutions have had to address in introducing new systems. ODP and ESL projects, along with EDD projects, have learnt important technical and management lessons concerning digitising materials, negotiating copyrights, cost recovery and academic culture.

Sufficient progress appears to have been made with ODP and ESL on the technical and management issues involved to make possible a valuable contribution to relieving pressures on key course texts for students. While at least one EDD project has provided tangible benefits for customer libraries within the lifetime of the programme, the EDD projects remain the most technically ambitious area of eLib and, despite important progress, EDD continues to pose significant challenges of service and organisation if its promise is to realised.

Electronic journals, quality assurance and pre-prints

The various EJ projects provide a rich variety of approaches and strategies and together provide a basis for scaling up initiatives or launching new initiatives based on their experiences. Four generic issues have been addressed by most projects in this are – publishing and associated charging issues; effects on scholarly practice; authors and authoring; and product and technical issues.

Most of the EJ projects have struggled with problems of sustainability beyond the programme funding: Many have pushed their innovations a little beyond what current academic culture is ready for or what publishers can realistically commit themselves to. Nevertheless many products were received enthusiastically by sections of their target populations and will doubtless re-appear in some form when some outstanding issues have been resolved. While wider culture change will make EJs more attractive to academics over time, the institutional issues involved in sustaining EJs are not likely to be resolved quickly or easily.

Access to networked resources

The ANR projects mainly involve subject based servers plus a meta-project which supports subject based servers and works towards systemisation of network navigational systems.

While all projects in this area are now providing functioning and well used services there still appears to be much to be accomplished to achieve the integrated resource discovery service originally envisaged. Further progress in this area building on the very real achievements to date will largely depend on broader strategic JISC decisions concerning national resource discovery policy.

Hybrid libraries and clumps

The eLib Phase III projects which began in 1998 mark a new departure in programme learning with practical attempts to integrate different socio-technical elements in working institutional and disciplinary contexts. Establishing working models of hybrid libraries requires effectively engaging with user cultures and sorting out perennial issues with digitisation and copyright, as well as creating effective interfaces to systems. Clump projects face specific technical and organisational challenges in reconciling information strategies across multiple institutions.

Generic issues in programme progress

Project and programme performance continues in this third year to be vitally effected by a range of process issues around staffing; management of consortia; funding; business plans/exit strategies; and evaluation. It is notable that all of these re-appear in familiar guises with the Phase III projects -notwithstanding considerable programme learning and increasingly sophisticated responses in terms of programme direction – indicating their endemic nature and systemic challenge for higher education.

Training and awareness, engagement and cultural change

The T&A projects continue to make a contribution to the overall change process in HE but the programme as a whole lacks institutionally embedded training strategies. To a large extent this can only be addressed through more systematic institutional engagement of the type initiated under Phase III. Significant cultural change will continue to await real material changes in structures and processes as well as thorough going institutional engagement and staff development at the level of individual HEIs and disciplines.

Conclusions

While overall judgements on the outputs and impact of eLib, and, indeed on its value and worth, will await the independent summative evaluation planned for 1999, it is clear, from this review of annual reports, that eLib has not only fulfilled its Follett brief but has already critically affected the ways in which the community understands the opportunities for the development of HE library services.

1. Introduction

1.1 Introduction

The syntheses of annual reports in the Tavistock’s evaluation services to the eLib Programme were originally planned to address the needs of programme participants to enlarge their knowledge of the programme and alert them to areas of potential common interest. This report is intended for use as a means to communicate with a wider, external audience and tries to give such readers a sense of where they could find learning about particular issues of interest. The report aims to provide a flavour of the way the programme learnt through the experience of implementation and practice, and locates it in an action research and learning context.

There are a number of difficulties with a report of this nature. For readers with specialist interests in specific domains summary materials are likely to be found superficial, while for the general reader any technical detail at all may be found to be too detailed. While intended at the outset as a communication across the programme and to the wider community, these reports have served first and foremost as evaluative reports for FIGIT and now CEI.

Further, the great value of this annual reporting system has proved to be in the immediate project and domain effects of being required to reflect on and reconsider project direction in the light of learning on at least an annual basis: The process of annual reporting has had a much more important formative effect on the programme than the actual resulting synthesis of annual reports itself.

1.2 Starting Points

In setting out what has been learnt within this programme and what can be learnt from it, it is useful to remind ourselves of what the initial starting points of the eLib programme were and to map against these the progress that has been made in exploring these. There were inherent strengths and limitations of the programme, embedded in its history, as first FIGIT and then CEI itself set out to provide a practical specification and operationalisation of the Follett Report, which we attempted to describe in our early Policy Mapping Study. In the Policy Mapping Study (PMS) we identified two key differences between the Follett Report and the eLib workplan - one offering a broader perspective than that offered by Follett’s report and the other providing a less comprehensive institutional focus:

Follett was concerned with library provision, with a remit to examine library and related provision in the light of the growth of undergraduate student numbers, changes to the demographic profile of students and new styles of course provision and learning approaches. Libraries were seen to be facing a crisis of spiralling costs - particularly of periodicals and specialist books - and pressure on space. Cost savings in the short term would be impossible to achieve but an investment to provide for long-term cost effectiveness might be sought in deploying it.

FIGIT while accepting and addressing these (the PMS confirmed that every substantial task concerning the use of ICT in library provision identified by the Follett report had been addressed within the eLib programme) went further in more comprehensively addressing the issue of provision of electronic information, leading to considerations of uses of ICT directly in scholarly practice and in teaching and learning - ‘not new ways of performing old functions but ....the development of new modes of action.’ (And it is no coincidence, as an old party member might say, that eLib is now steering by a committee styled the ‘Committee on Electronic Information’).

As far as the use of ICT was concerned, the Follett report had two main themes:

First, the need for deploying new developments which could help increase physical storage space and increase efficiency, speed and flexibility of document access and delivery.

Second, the need for better integration at all levels:

- at the hardware and software levels to increase interoperability, efficiency and usability;

- at the level of the library information management strategy to integrate library provision within a coherent institutional approach to management, and to teaching and learning;

- at the level of the whole institution through greater collaboration and sharing of resources across whole institutions.

In could be said that eLib has only been (and could only have been) ‘working towards’ this type of institutional impact, and, as we will see later in this report, the recent launch of the hybrid library and clump projects reflects the job of ground clearing that was required to even begin to achieve such institutional ends.

For example, the 1996 Edition of the eLib programme pack stated:

‘The main aim of the eLib programme, through its projects, is to engage the Higher Education community in developing and shaping the implementation of the electronic library.’

And that the chief sub-aims were

a) operational - in terms of developing the organisation and delivery of library services based on electronic systems; and

b) cultural - in terms of shifts in attitudes and skills among librarians and their user communities.

But as we commented then

‘In this sense, the programme is moving towards the electronic library (though with an expectation that only some components of the concept are relevant and feasible in the foreseeable future.)’

and also that:

‘[while the] consortium structure supports networking across institutions regarding information services it also appears that some projects will be limited in their ability to exercise leverage upon their own institutions ... Many projects are (for good reasons) too small, too brief, and have too few staff to be able to address the full set of change issues implied by their work or to be able to secure deeper institutional commitment.’

Other limitations noted at the start of the programme, such as the investment required in project management of consortia, pose questions about programme management. For example, how far were weaknesses identified at the early stage compensated for by action at the programme level or developments elsewhere? Was appropriate support available to help projects plan adequately? Were the right structures developed to enable projects to learn from each other?

Some of these questions will await for definitive answers from the summative evaluation, such as:

• How appropriate was the scale of projects and what problems arose from sheer numbers and in scaling up?

• How successful have the project selection, management, steering and evaluation mechanisms been?

• How adequate were the communication, T & A, and dissemination mechanisms?

However, even at this stage it is possible to make some judgements on to what extent eLib has achieved its programme aims: In our view the third year annual reports show, when combined with the plans of the Phase 3 projects, that eLib has provided the basis for the organisation and delivery of the library systems based on electronic systems required by Follett. And further, while individuals may judge for their own location how deep the mobilisation and catalytic effects of eLib have been, it would appear to be a common place - at least within the library community - that eLib has profoundly changed how actors perceive the current situation and imagine the future.

1.3 Constraints and limitations of the report

As we have said the final summative report will be prepared by an independent group of evaluators. This synthesis of third year annual reports is one input into that final reporting which we anticipate will also cover the very many successful outputs of the programme in terms of products and services as well as process issues which will cover our own role.

However, the set of Annual Reports provided to us was not a complete set from all projects in the programme and must be seen as a partial record of this third year. Some of the reports received are final reports, others interim reports if finishing at year end or receiving on-funding.

A number of projects have responded to the invitation we issued in the guidelines for their final year to reflect on what they have learned over the life of their project, offering a broader scope to this paper. So this paper contributes to an overview of learning over the three years of the programme as well as of this year’s activities. However, this paper does not detail all the many achievements in terms of outputs of the programme and there are important gaps left in the account as a result of missing reports. Consequently, we are aware that our account may fail to do justice to many actors and projects and trust they will understand the constraints we have operated within.

1.4 Structure for this report

The first report in this series, Synthesis of eLib Annual Reports January 1997, followed the framework for reporting we provided to eLib projects. Last year’s synthesis of annual reports was mainly structured around substantive developments in technology in the different domains (ODP, ESL, etc.). At the end of the programme’s third year many projects are coming to an end and future directions for the programme and the field more generally have either become clearer or the dilemmas of choosing between different options have become more apparent. This has had two consequences for our reporting framework:

First, at this stage it seems important to review activities and progress of projects against the original aims for the domain which they were commissioned to develop and examine how far these have been modified in the light of experience or superseded by external developments in HE such as movement of publishers around cost-recovery issues and copyright and changes in the technological environment.

Second, a balance needs to be struck between learning both from the various domains on the one hand and on the other from generic thematic learning. Learning both from this programme and associated developments has sometimes changed the way different domains are perceived and relate to each other.

Each content area therefore addresses a set of key themes, such as copyright, publishers, the technical tasks and so on. These themes at times occur in more than one place in this paper. Such themes are often relevant to the whole programme but may be particularly germane to that subset of projects, which thus provide a material context in which they may be discussed.

We have used the annual reports from this third year to reflect on the programme as a whole, examining project activities and progress over the past year towards objectives (i.e. not project management issues). The content of domain sections (Section 1) will also in some cases draw on material from evaluation reports and annual reports from previous years as well as from the Policy Mapping Study, the supporting studies and in one instance an article by the Programme Director. This additional material will be used to contextualise the material from the third year Annual Reports, which remain the chief data source for this year’s synthesis.

Since the focus of this report is on learning from the programme, the domains are clustered together with others where learning has heavily overlapped. Thus On Demand Publishing, Electronic Short Loan and Electronic Document Delivery are considered alongside each other for the purposes of unpicking some issues of more general interest in the programme, since some of the same lessons have been explored in all three domains. Similarly, learning from the Electronic Journals Domain is considered together with that emerging from Quality Assurance and Pre-Prints.

Programme and Project Management issues relevant to all three phases of the programme will follow the domain sections.

Two domains are treated in a different way. Previous programmatic learning is reflected to a considerable degree by the first reports from projects commissioned in the third phase of the programme, the Hybrid Libraries and Clumps projects, which had not been envisaged in the original programmatic or domain objectives. We therefore use a different structure for synthesising the learning captured by these domains.

Projects in the Training and Awareness domain are also considered in the light of continuing outstanding developments required. Both individual behaviour and institutional needs are addressed in the penultimate section before we present our conclusions.

As in earlier syntheses, readers are strongly urged to refer to the project reports in full where they have a particular interest; these are generally available on project web sites.

2. On Demand Publishing, Electronic Shortloans and Electronic Document Delivery

2.1 Introduction: Scope of the domain

The ODP and ESL project learning highlights many social, cultural and institutional issues and the progress that has been made within participating institutions while some of the technical and management lessons involved in digitising materials, negotiating copyright, cost recovery and academic cultural issues involve shared learning with EDD projects. We therefore discuss these three domains together.

ODP encompassed networked access to, and on demand publishing of, learning material or text, particularly those held in multiple copies, and also collated documents that could be flexibly used or customised by readers. Follett recommended the funding of demonstrator ‘on demand’ projects mounted at one or more host universities which would also develop appropriate copyright payment systems. This was adopted by FIGIT, which went further in setting out to explore electronic access and delivery mechanisms for high demand materials in short loan collections.

In the three ESL projects there was a focus on copyright issues through investigation of third party rights management, generic rights management systems and prototyping multi-media electronic systems.

The ODP projects aimed to develop demonstration services in provision of materials for students, investigating not only copyright but also charging mechanisms as well as developing appropriate materials and running digitised databases. In our Policy Mapping Study we commented that ODP projects were not so much offering new services as ‘but rather experimental services/scoping studies’. These would need more work on integration with teaching strategies and library reorganisation and with publisher supply strategies than seemed likely to be achieved through this set of projects. This proved largely true in practice, and in their latest Annual Report ERIMS comments:

‘The initial aim of implementing a demand-driven, networked electronic service over four university sites was tempered by experiences. There was an on-going struggle between the need for a service to answer current resource provision requirements and what in reality was a research project to develop and test a pilot system to access digitised materials.’

The aim for ODP and ESL was to provide tailored, targeted ‘resources for courses’: at its most sophisticated, the vision incorporated multi media resources as part of ordinary library provision, including computer-based learning materials addressing common curriculum elements across a wide range of courses and institutions. However, this required deployment of courseware and electronic sources to be integrated within broader pedagogic strategies and a more comprehensive provision of electronic texts and images than was possible within the term of these projects.

EDD projects were among the most technically ambitious of the eLib programme but also posed significant challenges to service and organisation. The complex technical development required

seemed to sit uneasily within the programme from the outset. In the second year’s synthesis we reported that "a shared view is that the tasks taken on by projects were more complex than had been imagined, and that closer cooperation and convergence had the potential to benefit all parties". In particular, this year saw the merger of two projects. We will touch on some of these issues at the end of this section but they are also taken up separately under the Programme and Project Management section.

2.2 Learning reported

Annual Reports to August 1998 available to us included:

• in ODP: Edbank, Scope, ERIMS and Eurotext

• in ESL: PATRON, ACORN, and ERCOMS

• in EDD: EDDIS, Seren and LAMDA

We also drew on previous Annual Reports and our own syntheses of these, and the supporting study. Some annual reports, e.g. Scope, provided an overview of learning over the three years. Others were more limited in their reporting.

The two most commonly shared constraints faced by projects across these domains in achieving their aims had to do with:

a) copyright clearance and cost recovery

b) digitisation of text

c) delivery, in ODP/ESL as concerned the restrictions on access by students, and in EDD as concerned the complexities of the software development process required.

Also significant for ODP and ESL were:

d) pedagogic issues involved in academic collaboration with librarians.

2.2.1 Copyright clearance

Copyright clearance problems were encountered early in the life of projects and overall were found to be very time-consuming for project officers. Most ODP/ESL projects opted for clearance for the term of the project only, but Scope negotiated longer-term arrangements. (Scope, for example, based negotiation on the terms of a model contract which includes definitions, warranties and indemnities on both parties and a description of payment mechanism.) To date, larger publishers have been more co-operative than smaller houses, journals are much easier to clear than books and electronic copyright is not always assigned by authors to publishers, making publishers wary about indemnities. But Scope’s final report suggests some step change in publisher’s orientation may have occurred since the early days of projects. One associated difficulty in ODP/ESL was knowing early enough what academics wanted clearance for.

ERIMS final report identifies the wider difficulty of achieving culture change throughout the chain of activities:

‘We found the requirements for setting up a framework for On Demand Publishing within the existing academic and publisher settings were difficult to achieve. The timing for submission of reading lists, the format of these lists and the need for updating them was not easily reconciled with the lead-time required for copyright clearance and scanning.’ (p.19)

ERCOMS addressed the development of an electronic copyright management system (the main aim of the project). This was to build onto a WWW server and comprises modules for user registration and access control, WWW browser, usage statistics collection management system, reporting and accounting. A separate PC-based instrument was to be developed for managing copyright negotiations with publishers. However ERCOMS states in their 1998 Annual Report that the system is still at the prototype stage. In the longer term, sector-wide negotiations are likely to be a more efficient way forward than each project negotiating with each publisher separately over each separate document.

PATRON has identified four key factors in rights management: the market sector and type of use; the region; the type of delivery; and the levels of security offered at various stages. Rights issues are most complex in relation to multimedia as reported by PATRON, including copying, issuing copies to the public, performing, showing or playing the work in public, broadcasting or distributing the work via cable, making an adaptation of the work. Dance material proved especially difficult to clear. Cost recovery schemes floated were often linked to authorisation procedures and their security. SCOPE appears to have met the publisher’s requirements for security levels satisfactorily but concerns about part-time and off-campus students, whose needs are most acute, remain. And one Scope partner retired from the consortium as a direct result of the security and authorisation requirements of the publishers.

Within EDD, LAMDA by contrast were able to work within the existing copyright legislation to share journal resources across a distributed consortium, delivering direct to customer workstations for printing for requester. LAMDA charges customer institutions £3.60 per successful transaction. Charging students however, as seen in evaluation reports for projects within ODP and ESL, is highly contested, even though students currently pay for photocopying and in some cases print-offs are cheaper. Furthermore, academics preparing course material want to provide materials to their students selected on criteria which are unimpeded by copyright constraints, in book as well as journals. In some projects substitutions have been offered, but they were not well received.

Within ODP/ESL various cost recovery mechanisms have been experimented with, including: pay to view; pay to print and site licences. Edbank considered pay to view but their publishing partner, Routledge, eventually came down on the side of site licences which are, in any case, cheaper for HEIs to administer. Most projects were charging on pay to print basis. ERIMS has used a payment model which passes royalty costs on to end-users. In SCOPE member libraries will be charged copyright fees for material supplied to users, as on the LAMDA model. Some Scope member institutions will pay these fees from the library budget and some from departmental budgets but others are considering passing copyright fees to students when they trial on-line delivery.

Academics may author their own student coursework packages and mount them on the web as an alternative to charging for copyrighted extracts, if they have the time and energy. Eurotext attempted to develop model home pages for certain courses which would be available on the Web to include course overview, reading lists with links to full texts, exam papers, lecturers’ comments etc. eOn also planned to develop an initiative whereby lecturers mount in-house course materials on the WWW.

However, materials for which rights are assigned comprise so much of the desired resources for these domains that solutions will need to be further explored with publishers. Publishers need cost recovery mechanisms which match alternative supply routes and use will have to be heavy to justify these demands. Over time it may be possible to negotiate costs of clearance to an acceptable level. However, the length of elapsed time involved currently creates prohibitive delays.

2.2.2 Digitisation

The cost of having materials digitised was consistently reported by these projects in the third year as in earlier years, as a major area of difficulty. As in other domains such as Digitisation and ANR, achieving critical mass of digitised material was especially problematic. By the end of Year One, EDD projects also reported major difficulties with scanning. LAMDA, having from the outset opted to use off the shelf hardware and software introduced new driver software from RLG and introduced Fujitsu scanners at its keysites which largely overcome these problems. However in ODP, most projects reported that digitisation costs could be considerable if material was not available electronically; almost always the material requested was not. The scanning equipment itself was not a major capital investment but the staff time involved in proof reading where forms of OCR were used had not been fully appreciated. Although the technology is available and dependable, it is not viable for a single institution alone. Joint commissioning of digitised texts would provide economies of scale. The service provided by HEDS enabled the projects to test out other aspects of the design but the digitisation issue was widely seen as an obstacle requiring step changes in technology before developments in these domains can be implemented more widely.

2.2.3 Delivery

Even in pilot sites students may have restricted access, though generally access is improving, it still varies widely. Dial-up access via a modem for distance learning students is not usually possible yet because of security issues. These are currently being investigated. Computer literacy among students is increasing and most requests for help were for help in printing. In terms of multi-media, Edbank found that digitising multimedia material was possible but there were downloading problems. PATRON reported that the tools available to the user can significantly extend the ways in which the resource can be used. ERCOMS’ prototype management system requires a secure Apache web server and a web browser with Java software.

EDD faces more profound structural challenges with publishers developing their own web sites and offering academics the opportunity to retrieve material directly, sending payment electronically and bypassing libraries.

2.2.4 Pedagogic Use

Third year reports highlight evaluation findings that academic enthusiasm for developing resources for students is mostly low. It may be that institutional pressure across the sector arising from the TAE and other steps on teaching and learning will be more successful in inducing academics to adapt their pedagogic practice. Pedagogic issues mainly focus on academic fears of spoonfeeding students by providing limited text in a pre-packaged form. Pressure on academics to include substitute texts due to copyright difficulties further decreases academic enthusiasm to invest time and effort. Projects need to start with the materials that are already in demand in hard copy form if they are to make progress with academics.

Fears that students would not pay for such resources seem not to be borne out by the evaluations. ACORN reported "an unusual result of the User Feedback was that one of the key uses of ACORN in both semesters was for wider reading." A general finding among these projects was that students who buy books and copy materials currently are likely to accept the burdens of payment. On the other hand the ethics of charging them are hotly debated by teachers within HEIs. Institutional support for academic staff involved in projects was very low in more than one institution, according to one supporting study, and some academics who are keen to develop electronic course material for students would prefer to focus their efforts on a WWW solution.

2.3 Achievements and looking to the future

Sufficient progress appears to have been made in ODP/ESL on the management and technical issues involved to enable the domains to have a valuable contribution to relieving pressures on key course texts for students. Expertise gained through Scope and Phoenix will be brought together within a newly funded project, HERON to exploit this. While the monetary costs involved in copyright clearance can be negotiated over time to a suitable level, the time delays are prohibitive. HERON will seek to build a databank of reusable, pre-cleared material which can be drawn upon by those HEIs which are ready to pursue this option. As we commented in the Policy Mapping Study, this will need ‘a high level of institutional commitment to change, central involvement of lecturers, and assurance of adequate, available equipment (PCs, printers) for students’. The costs imply a narrowing of the material which can be included due to digitisation and copyright difficulties. Taken together these factors suggest relatively slow uptake of such schemes in the near future but significant gains can be achieved in he medium term.

LAMDA has been making a contribution to Electronic Document Delivery within the life of the programme, providing tangible benefits to the 50 universities on 69 sites which are customer libraries. Use of the Union list of upwards of 1,200 journal titles, accessed through COPAC has increased competitiveness despite shortcomings in the service. A z39.50 interface is being developed by LAMDA alongside other planned performance enhancements as the project is being launched independently beyond the life of the programme.

Other EDD projects like EDDIS and Seren have made progress with software which will develop electronic document delivery capability. Seren sees itself as capable of supplying items at a lesser cost than a request to BLDSC. EDDIS has transitional funding for a further year while negotiations between a consortium member and the commercial partner determine exploitation arrangements. The support project to this area, FIDDO, still has some time to run (to May 1999). FIDDO staff profess themselves to be optimistic about the future for the area: ‘Notwithstanding BLDSC’s dominant position within the UK document delivery market, it is FIDDO’s view that networked end user-oriented full-text systems will achieve an important position in that market within a few years.’ However, for the moment their view is unchanged since the previous year: ‘that trials of electronic document delivery systems were impractical in the context of an ILL facility.’

3. Electronic Journals, Quality Assurance and Preprints

3.1 Introduction: Scope of the domain

Electronic Journals were seen by Follett as supporting new developments in academic practice and as a potentially radical innovation. The report argued that their status, credibility and their likely threat to conventional publishing should be explored through the funding of a limited number of Electronic Journal development projects. Within the eLib Programme this area covered both substantive journal services and generic system architectures. The activities proposed possibly extended original policy conceptions in moving towards changes in academic practice and modes of knowledge production. In Quality Assurance and in Pre-prints, peer review came under consideration more broadly in the programme.

At the outset of the programme we commented that the Electronic Journal domain contained a rich variety of approaches and strategies: from electronic text to full multi-media: from broadcast to communication; from systems development to action research. Not all projects had a disciplinary strategy for exploiting EJs as a new medium of scholarly exchange. However the projects together could provide a basis for scaling up or launching new initiatives based on their experiences: These were sometimes flagship disciplinary initiatives consisting of full journal environments linked to an ANR gateway, digitised resources and union catalogues. (Social Science, History, Chemistry and Medicine).

The activities undertaken by these sets of projects included producing and publishing journals and launching new electronic services.

3.2 Learning reported

For the third and final synthesis from the evaluation team the following Annual or Final Reports were available:

• For eJournals: The Electronic Stacks Project; Open Journal Project; Sociological Research Online; Internet Archaeology; Electronic Reviews in History; Electronic Seminars in History; the CLIC Consortium. A Flagship Chemistry Electronic Journal; Newsagent for Libraries; Parallel Publishing Transactions; The Electronic Journals and Learned Societies

• For Pre-Prints: Education Online; Formations; WoPEc

• For Quality Assurance: Espere

The idea behind this domain was to look at what could be achieved through developing new electronic media which would play a similar role to that of traditional academic communication through print but with additional enhancements. Enhancements would include developing access to ‘pre-print’ material through the working papers to be made available in the Pre-Prints area as well as enhancements to text such as multi-media. The sets of projects in eJournals and preprints, together with those in Quality Assurance, were seen as vehicles for the exploration of quality assurance issues of which print journals have traditionally been the guardians, in exchange for commercial rights to exploitation. These project areas therefore had considerable implications for the role of publishers, both scholarly societies and large-scale publishing houses, who were variously involved in consortia arrangements, as partners or associates.

The starting points for individual projects were on the one hand ambitious about what could be achieved and on the other saw their role very much in terms of traditional print products. Electronic Stacks Project comments, ‘This project started in the belief that electronic academic publishing is going to displace the traditional academic journal and book publishing.’ The Open Journal Project quotes from its own mission statement that they aimed ‘to build a framework for publishing applications enabling journals on the Web to be interlinked in ways which build on the traditional qualities and identities of the journals.’

3.2.1 Publishing and charging

Four generic issues had to be addressed by most projects in this area:

a) publishing and associated charging issues

b) effects on scholarly practice

c) authors - and authoring

d) product and technical issues.

Projects in these domains generally included publishers in their consortia arrangements, though caution in relation to cost recovery often inhibited their active involvement. Small-scale and scholarly societies were those most often involved with limited impact on publishers as a potential stakeholding group. However, one or two larger projects tried to involve a range of large-scale publishers.

Initial activities in the first year included: attracting subscribers through various promotional activities; forming editorial boards and establishing procedures to support the editorial process; drawing up agreements to cover liability for work published; generating material; engaging in debates about issues such as subscriptions, management of copyright and intellectual property rights, refereeing, etc. During this phase there was widespread acknowledgement of the complexity of the issues of copyright and libel, increasing recognition of the management aspects of launching a journal, such as ensuring a supply of articles and handling the ongoing editorial work; development of appropriate work procedures and mutual understandings as well as technical aspects. Complexity characterising software development environments was often highlighted while at the same time close co-operative relationships between the various parties was seen to be a precondition for development. The preliminary finding was that the costs of running eJournals was not lower than those associated with traditional print journals.

By the beginning of the second year EJ projects perceived the need to explore charging mechanisms and revenue-raising and to undertake market testing. Publishers were concurrently developing their own relationship to the technology through website ordering and electronic versions of existing journals. Their relationship with eJournal projects within this programme were informed by this context. Thus the Open Journal Project commented at end of Year One: ‘Another gap is that between traditional publishers, who own the print journals, and the new generation of would-be Web publishers. A publisher can be both at the same time, but the gap is still there. Traditional publishers are naturally conservative and, under pressure to preserve commercial profitability, are faced with a new medium that has not demonstrated the capacity for sustained revenue generation. Such publishers are not inclined to be ‘open’ in the same terms that support the Web philosophy: the publishing process itself is staunchly resistant to any apparent form of standardisation. It is against this background that the project has to involve publishers in developing new products that may, or may not, be consistent with their existing commercial strategies, and for which the demand from the academic community is unclear. An added dimension for the project is the extension of the Web culture to its underlying link culture.’

However, while many new products and services were delivered within these domains, constraints were identified in relation to academics and to publishers, two major stakeholders in these projects. The cultural constraints had implications for the scale of cost recovery which might be envisaged, and thus for the commercial viability of publishers involved.

3.2.2 Scholarly practice

The scale of the changes in scholarly practice which might be involved and a sharpened appreciation of the preconditions for their success often formed the first round of learning of these projects. While early advances (soft gains) were noted by eJournals in disciplines new to the technology, in social science, archaeology and history for example, where academics were more familiar with the technology constraints in relation to scholarly practice began to be more clearly identified during the second year of the programme. For example, in organic chemistry CLIC noted that authors drawn from a more technically-oriented community were less ready to use text enhancements than they would have envisaged.

Some projects required academics to be behaving in different ways early in the project’s life in order for demonstrators to be tested. For example, in DeLiberations:

‘The key to the success of this magazine is the active participation by the ‘readers’ in the production of the magazine through discussion and more formal contribution. The aim is that the HE community should adopt DeLiberations as a means of ‘talking to’ each other; in contrast to ‘commenting on‘ articles. The conclusion is that new technology of itself does not automatically alter people’s behaviour, but that the power of email and the resources of the WWW can be incorporated into existing relationships and can help to create new ones.’

The Open Journal Project found that motivating users to comment was not easy. Towards the latter half of the project’s life, progress in the project involved hiding many of the technical features from users in response to earlier feedback.

Certainly new relationships and even new communities have been facilitated within the life of these projects. Sociology Online and Internet Archaeology, for example, have demonstrated how eJournals can facilitate academic communication across national and traditional boundaries. The projects’ achievements in terms of products and rising readership figures tell a story of increasing uptake. However, none of the journals saw themselves as viable within a purely commercial environment, when discussing their exit strategies at the end of Year Three.

The ways in which academics use journals appeared not to have altered much over the life of the programme. Most want to run off print copies for annotation rather than using them online. While there are reports of Departments giving up paper copies of journals, most want their University libraries to continue to provide space for hard copy for storage and archive purposes. However, it is probably reasonable to speculate that as academics across disciplines come to internalise their learning they will be more open to extending their use to enhanced capabilities of the media. This will depend on the necessary preconditions with respect to authorship, publishing and so on being met.

For academics to exploit the opportunities afforded by the technology through these projects often involved behaviour change which there were not sufficient incentives to bring about. Concerns about quality assurance and investing in authorship surfaced early on and were not dispelled over the life of the programme. As the programme developed the enthusiasts became more reflective about the cultural change which would be required for increased uptake. By the end of Year Two, the task of achieving sustainability beyond eLib funding was a shared concern of these projects: for some the task ahead was to explore pricing strategies and to investigate charging mechanisms; others called for corporate change, risk and new investment strategies by commercial partners, a call which was to go largely unheeded. Their final exit strategies at the close of Year Three were considerably more subdued, often they seemed grateful to survive in more attenuated form within scholarly societies.

3.2.3 Authors

In order to provide a degree of usefulness which justifies deficiencies in usability, the technology must be providing clear additional gains. In the Open Journal Project, an important focus was on citation links:

’A critical issue the project must solve is that of creating high quality links in volume.. The project has demonstrated the application of links on a large scale, but not yet the effectiveness to produce only useful links on the scale that commercial link publishing will demand. …First tests with biology links have begun to indicate what a quality link is, but it can differ for a given user or group of users.’

The Electronic Stacks Project also saw implementation as a medium term development: ‘Although a version of these tools is installed at various sites.... wider application is likely to require commercialisation of the tools ... (which) is not imminent.’ The link service approach of the Open Journal approach poses difficulties for Web publishers used to maintaining a core of data on static HTML sites.

Gains to academic authors, when measured against the effort involved may be less than for academic readers. Authors are commonly perceived by these projects as seeing electronic publication as too ephemeral to invest in. This often results in low numbers of contributions experienced by several, but not all projects. Internet Archaeology addressed this fear of authors by joining with Archaeology Data Service subsection of AHDS to provide long-term accessibility of articles. Particular problems faced journals who wished authors to invest in text enhancements, as at CLIC or multimedia as in Internet Archaeology. In CLIC, where enhanced features were seen as desirable, these tools required development of mark-up languages for authors, after early attempts based on VRML were found to be too difficult for authors to use. Several projects settled for the more limited scope afforded by PDF, which became accepted as the standard for the sector over the life of the programme’s first phases.

Both Internet Archaeology and CLIC invested in author’s use of enhanced text, in CLIC by providing in-house advice and support, strategies which met with some success. As a result they have been able to demonstrate the added attractions this provides. Sociological Research Online reports a similar positive response to encouragement and suggestions to authors. However, the personal approach has been an additional burden for the project teams and has implications for scaleability. Possibly such author use of text enhancements will depend on adoption on a discipline-wide basis for wider take-up. Its usefulness when balanced against the effort involved has not been established by these projects at this stage. In terms of readers, CLIC in particular has registered disappointing evaluation results from readers for the enhanced features. However the Royal Society of Chemistry, the future hosts of CLIC, intend to feed the experience into their design of other journals; if followed through then it would result in a step increase in use of the media within the discipline.

In general author experience within these sets of projects has been very mixed, depending on opportunities for authoring within the different disciplines. For example, electronic communication of research results within the medical sciences is time-sensitive to a degree which motivates electronic communication. Electronic Seminars in History reports a long waiting list for authors to get into print in the discipline driving publication.

3.2.4 Changing products and technology issues

The issues of the changed nature of the products developed within these domains as compared to traditional products and their relative take-up within different scholarly communities is intertwined with technical issues in supply and delivery.

Access and delivery problems dogged the projects early on, the usability issues tending to obscure the more central usefulness issues. Even up to the end of Year Two, our own thinking about this set of projects continued to be tied to the notion of eJournals as alternatives to print journals rather than seeing the interactive possibilities provided by eJournals and preprints together as additional to print journal capability.

The whole idea of an article is transformed by electronic publication. Whereas in print articles were of relatively fixed length and type, in the electronic environment they are quite different. They can ‘consist of databases as readily as extended multimedia essays’ (Open Journal Project). However, both authors and readers need the tools to use these enhanced features in compatible ways. Unfortunately these problems were not in sight of resolution within the term of these projects although considerable progress continues to be made and is evident in the third round of annual reports.

Readers access to the developing electronic products and services remains mixed according to the third round of reports. Sociological Research Online’s early evaluation report turned out to have been unduly pessimistic about delivery platforms for academics. CLIC reports that many central and departmental libraries are now buying computers specifically for electronic journal subscriptions which are more able and willing to deal with electronic resources like CD ROMs than at the start of the programme. Leeds University Chemistry Department has given up its paper based journal subscription with a saving of £30k per annum

Use of the medium for peer review may also be considered as a use. CLIC reports that the RSC is now using web-based referee forms for all peer reviewed journals which has become the preferred method for reporting, with associated cost saving. Electronic Stacks Project reports that the software has been developed for administration of submission and refereeing processes but is not yet in operation. ESPERE sees the problem of compatibility of platforms as an outstanding issue to be resolved.

The technical solutions and fixes when they come appear to be partial and transitory. They are also embedded in a range of professional and organisational issues which those involved in technical developments often underestimate or fail to take into account. So far, advances in technical innovation have not been matched by perceptions of its practical usefulness balanced against usability.

3.3 Achievements and looking to the future

The search for sustainability highlights the perceived usefulness of the product to a set of end users. (Although as the Programme Director rightly points out, perceived usefulness will not overcome all the problems of sustainability.) The culture change required to make some of these products and services viable was lacking but even the involvement of users in the development and design of products was weak. For example, the Open Journal evaluation reports that there was not enough user element in the design of the project early on. Sociology Research Online concurs: ‘although the project has fully met its main objective, attitudes within the academic community towards e publishing have not yet changed sufficiently to ensure its immediate success as a self-financing publication.’

There have clearly been problems in developing standards across the sector, especially in relation to enhanced text. Project final reports are littered with comments such as ‘ahead of their time’. In some ways the most innovative and speculative area of the programme, it would be surprising if the enthusiasts had not sometimes buried themselves in the work of addressing prototyping issues. It may be disconcerting to confront the more sober judgements of programme managers when funding Phases 2 and 3 of the programme, compared to some of these early exciting efforts, especially those in Pre-prints. Iconoclasm and exclusion of pragmatism may be the cost of stretching the boundaries of what is possible . Project managers were in some cases able to comment wryly on the resulting impact on sustainability beyond eLib funding in their final reports. While in some cases the issues of usefulness to the target users were not adequately addressed early enough, in others the project was not scaled appropriately. -

The role of publishers within the medium is still not sufficiently clear for them to commit fully to the new products. While OUP effectively ‘gave up’ copyright in one instance, the circumstances were specific to a particular community. Electronic Stacks reports in their Year Three account, that publishers now publish free abstracts and authors offer these freely on their own websites. However, they are unlikely to cede larger market segments and their reach and professionalism cannot be matched by academics’ spare time and enthusiasm.

Many products were received enthusiastically by sections of their target populations and will doubtless reappear in some form when some of the outstanding issues have been resolved. Some of the products developed appear less ephemeral than others e.g. authorship in Sociological Research Online is accepted in the Research Assessment Exercise for the discipline. Such products will be increasingly viable in the future. One problem with eJournals, preprints, etc. is that while traditional ownership of products is threatened. Publishers continue to provide critical expertise in developing and promoting products. New ways of distributing and recovering the costs involved in epublications are not yet sufficiently defined. Most projects have transitional funding, temporary homes and mergers which will allow some continuity of learning. However many only expect to survive in reduced or curtailed form.

While wider culture change will make these products more attractive to academics, institutional roles required for such products to be maintained are not likely to be resolved quickly or easily.

4. Access to Networked Resources

4.1 Introduction

Follett noted that the proliferation of software tools and net applications that had been developed for network search and retrieval had proceeded largely on a voluntaristic basis and argued for a more systematic approach and recommended funding of a limited number of top level navigation tools to encourage the growth of local subject based tools and information servers. FIGIT aimed at systemisation through the development of navigational tools to encourage the growth of local subject based tools and information servers. The domain of ANR covered mainly subject-based servers and, in ROADS, a meta project which supported subject based servers and worked towards systemisation of network navigational systems - and subject based servers. CATRIONA 11 had stronger institutional links than the other servers, linked to a group of Scottish HEIs.

4.2 Learning reported

Initially there were ten constituent projects; annual reports from the following were made available to us for our third round synthesis: CATRIONA II; EEVL; SOSIG; ROADS; OMNI; HISTORY; Biz/ed; RUDI; ADAM.

The area of the programme was developed from the need identified in Follett to systematise the wide-ranging gateway options confronting academics when they undertook electronic searches. These often came in a range of different formats and sometimes required a range of delivery platforms which was confusing and frustrating, especially to media novices. There were concerns about the lack of coverage across subject areas represented in this set of projects and the possible problems of interoperability further down the line if the subject-based approach went too far. The programme managers were aware of the limitations of the approach and sought ideas for alternative services which might provide complementary explorations.

4.2.1 Technical tasks

In Year One we commented that projects in this domain were engaged in a similar set of activities, although the established nature of networked resources in some disciplinary areas had enabled some projects to focus more on refining the interface and facilitating the creation of new networked resources rather than on developing the services specification and technical capabilities of the service. One project, dealing primarily with politically sensitive grey material had found considerable copyright problems.

Unsurprisingly technical work dominated the early period of many of these projects' life, with users mainly involved in evaluation of usability, which is discussed below. ANR Projects were in Year One: undertaking user needs requirements for resource discovery and cataloguing; developing resource descriptions; providing a test bed for ROADS as part of piloting a prototype; developing training materials and conducting training; facilitating the creation of new networked resources; producing technical documentation. Successes in the first year concerned: achievement of targets within or ahead of scheduled timeframe; positive user feedback on interfaces; technical development, including, for example, the shift from a general purpose WWW server software to a custom-server optimised for delivery of template database. Encouragement of new networked resources, while an important aspect of their work at this time, was proving time consuming and difficult to assess in terms of usefulness.

The second year reports revealed significant differences between the subject based gateway projects in terms of their achievements and likely futures. These judgements reflected varying circumstances, including the nature and boundedness of the discipline or field and relative felt need for cross-disciplinary searching; staff attributes; synergy with other projects or developments elsewhere; the nature of the material, such as relative desirability of multi-media; and the stage reached in the project lifecycle. EEVL, for example noted that

‘It has become apparent that the needs of engineers vis a vis electronic information mirrors their needs for printed information. Bibliographic information, plus easy access to quality Internet resources, but access to trade literature is also very important to engineers, whether in print or electronic formats. The composition of resources in the EEVL database reflects this, and makes that database quite different in character to those of OMNI and SOSIG. We feel that a focused main service database, and the 'value added services' may be far more appreciated by engineers than a seamless interface to all the SBIG.’

It has to be said that EEVL's view of their own success was endorsed by a number of awards over the life of the programme.

However SOSIG spoke for all these projects and many associated with the programme: ‘A continuing source of concern for us, and confusion in the minds of users, has been the lack of clarity about the role of NISS and BUBL, in relation to the subject services.’

Most projects reported their second year as one of consolidation, three of them having moved to operational service rather than pilot demonstrator provision.

A main development reported on in Year Two was the integration of thesauri to assist search and classification systems being adopted to allow more efficient indexing and searching. In ADAM resource discovery and cataloguing were being carried out to ensure that records were indexed with controlled terminology from the Art and Architecture Thesaurus and the Union List of Artist's names and were classified using the Dewey Decimal Classification system. SOSIG used Universal Decimal Classification system to cover a wide selection of social science subjects. This allowed for more specific browsing and was designed to help manage the increase in the number of resources. A manual approach to resource cataloguing turned out to be very time-consuming and labour intensive; but the quality control aspects of cataloguing processes won user support.

Consequently it was envisaged that the emphasis in some projects would shift away from resource discovery and description and would move increasingly towards the quality control, classification and peer review functions in the third year. Granularity remained an issue at Year Two: some method for describing relationships between resources was seen as necessary to enable coherent cataloguing and efficient retrieval. One problem identified at this stage was that the structure of the IAFA/ROADS templates did not allow any provision for creating names relationships between resource descriptions.

From early in the programme's life ROADS' contribution to the design of the technical configuration in various subject gateways was widely admired and found helpful. This view was reinforced in the Year Three reports. However, these also show continued uncertainty about the way forward. For example, the stage at which to migrate their services to ROADS' solutions and whether to upgrade at each ROADS release is debated between and within projects.

The uncertainly is highlighted in Year Three reports, especially in relation to the choices about integrated search interfaces, where debates between z39.50 search interfaces and others such as WHOIS++ continue. Most of the projects felt they had delivered a valuable and valued service whose usage was increasing. But satisfying the needs of the disciplines with which they were associated had led them in different directions. The need for systematisation had only been met in compartmentalised parts. Explorations of reconciling interfaces had provided conflicting or ambiguous messages about the possible solutions which might be adopted. Some projects give the sense that they had done all they could in setting up a service within the given parameters but that time was running out for hard choices to be made. There was both some hope and some apprehension about JISC's anticipated intentions for the area which were due to be made known shortly. This story is taken up in the chapter on Hybrids and Clumps which follows.

4.2.2. Evaluation - usability and usefulness

In Year One most projects were involved in the development of a pilot service or demonstration which could be used to elicit systematic feedback from users. A range of different methods were used, including on-line questionnaires, focus groups and practical trials. Mostly, these activities identified modifications or add-ons to services, contributing formatively to the ongoing development of project activities, as in the following examples:

a) One project used a model of exploration of these issues which is noteworthy. They worked in close association with a cognitive researcher who investigated users' information seeking and information use behaviour and the cognitive processes involved in deciding whether or not to use information retrieved from the Internet. A preliminary finding was that users have developed a range of criteria for assessing the quality of the sources, which fall primarily into the categories of: authority, accuracy, currency and representation.

b) Another project was led to conclude that evaluating networked resources was not yet a staple part of the daily work of a set of respondents, even though most of them were information professionals. The project response was to include sessions relating to evaluation and selection strategies for networked information resources to their service.

c) Usage data in other projects confirmed that users were employing relatively unsophisticated search techniques and responded by providing more documentation and support as part of the service (e.g. user guide and email help desk).

Evaluation efforts at this stage were mainly focused on usability of the service but nevertheless taken together suggest more fundamental questions of ultimate usefulness. However, this learning was perhaps too widely dispersed for the subject-based projects to discern its generality at this stage. In their objectives for the year ahead none of them foresaw changes in direction nor any serious challenges to the vision of what they were trying to achieve. Only ROADS, from a unique vantage point, expressed a reservation about the nature of the software and the culture of the community in which it was to be utilised as a potential constraint on uptake. The understanding of the learning for projects perhaps reflects our own emphasis in this programme on formative evaluation and its input into ongoing development and planning. This area is the subject of a section within the programme and project management section of this report, which will use these examples as a point of reflection.

By Year Two the issue of usefulness of services being developed to the end-user was being seen as an important element balancing the drive towards compatibility. Sociotechnical issues involved in the process of development were also surfacing and sometimes indicated the limits of electronic communication. ROADS observes that

‘there have been and continue to be differences of opinion within the project about the mechanisms for internal and external communications, particularly the value of face to face meetings. In spite of the high degree of commitment to the use of email and electronic document and information transfer at all collaboration sites, there remain disagreement about how completely these mechanisms can fulfil all communications needs. It seems that three cultures are represented within the project: technical/development oriented, intellectual/research oriented, and service/user oriented. There is certainly room for some research into how completely current tools allow collaborative distance working within such projects.’

By Year Three the gateways were providing online catalogues of resource descriptions within separate disciplines. However despite agreed common standards, this was still a long way to the kind of integrated resource discovery service originally envisaged. The future appears to lie in providing quality control, classification and peer review functions. ANR projects are likely increasingly to form distributed 'clumps' with common standards and protocols providing interoperable services. The problem of coverage across disciplines is one which the new Phase III Clumps strategy aims to address through much broader subject groupings than targeted by these projects. It remains to be seen if this will work.

4.3 Achievements and looking to the future

The third year's annual reports reveal mixed feelings about the future, amid speculation at JISC's direction for a national resource discovery network. On the one hand projects report on the services they have developed and evaluation reports show increasing usage of these. On the other they are still debating the choices between matching the needs of their "own" user group or adopting what they see as unsatisfactory solutions which will provide more integration between them. For projects such as EEVL's award winning service, for example, a possible change at strategic levels away from subject-based approaches provoked dismay and disappointment.

Projects are also reflecting that they have had a short time in which to establish a stable reliable service and note the tension between this aim and that of incorporating new and exciting ideas in a volatile environment. A great sense of pride and ownership invested in the development of these services can be seen in the Year Three reports, which were seen as at least essential building blocks in the systematising of gateway options.

5.1 Introduction

While in Phase 2 of the programme FIGIT/CEI had sought to provide greater coverage of areas of exploration, to fill in the picture provided by the Phase 1 projects, Phase 3 marked a new departure in terms of programme learning. By this time many of the most critical lessons from Phase 1 projects were in, providing news about what could be achieved in the short to medium term. Chris Rusbridge, the Programme Director, observed that while ‘JISC's projects are concentrated at the near-market, practical end of the spectrum’ relating to its mission ‘to stimulate and enable the cost effective exploitation of information systems and to provide a high quality national network infrastructure for the UK higher education and research councils communities’, the Phase 3 projects nonetheless appear to mark a new practicality about how different elements of the socio-technical picture may be reconciled in the foreseeable future. The projects of the Hybrid Libraries and Clumps domains provide a new balance of different institutional and cultural realities with enthusiasts’ visions that it is important to invest in some of the speculative explorations included in Phase 1 projects and also a balance between long-term strategic visions and current practicalities.

In this section we see how the learning from Phases 1 and 2 projects has led to the adoption of these newly conceived areas, drawing on reports from the projects for the area, the support project report, MODELS: Moving to Distributed Environments for Library Services, undertaken by UKOLN and an article by the Programme Director in D-Lib Magazine

5.1.1 Rethinking Programme Direction - The role of MODELS

Supporting studies within the eLib programme are not the focus of these syntheses of annual reports. Nevertheless, they have contributed to the Programme's understanding of different aspects of eLib as well as to our own. MODELS has been a particularly successful supporting study providing workshops for programme actors around design principles and technical explication, which have facilitated new thinking about the programme and enabled practical decisions to be taken and implemented. The report from MODELS for Year Three outlines some of the parameter setting work undertaken over the last year, during which they have mounted two workshops: one, on A Distributed National Electronic Resource? took place early in the year; the second, addressing MODELS Information Architecture (MIA) took place in early Summer.

Like other supporting studies, MODELS' role has been more about understandings than practical outcomes. However, the provision of workshop settings for programme actors to come together to reflect on what they are learning and build new understandings should itself be understood as a significant service to the programme. In this instance, practical outcomes may be seen in the design and planning of the Phase 3 projects and in the understandings which underpin these, in which MODELS has played the leading role. While the development of the ‘clumps’ concept is cited as the project's most significant achievement, their role in creating problem-solving fora may ultimately be seen as having an even greater impact: ‘Several participants acknowledged that the influence of MODELS is more important than specific individual achievements. One noted that the project has been successful in making things happen.’ (MODELS Annual Report 1997/8) MODELS' contributions to the Programme include the development of MODELS Information Architecture (MIA), a model systems framework to manage services building on the shared understanding of organisational and technical issues involved in distributed library services. Other direct outcomes have included the concept of 'Information landscapes'; helping progress the Distributed National Electronic Reserve (DNER), initiating the Clumps projects, providing a profile of z39.50, initiating an Interoperability Focus and a Directory of z39.50 targets in the UK.

From our understanding of the Programme Director's viewpoint, MODELS workshops were important in helping the Programme rise above some of the more negative messages emerging from Phases 1 and 2, especially from projects in the ANR, and revisit JISC's own mission statement in order to re-energise and rethink programme direction. (When we come discuss evaluation in the section on Programme and Project Management, we use this as an example of ‘double-loop’ learning - learning which not only helps reflect on how to do the same things better, but which helps reflect on alternative ways of doing things.) Those of us familiar with programmes which fail to go through this process in a timely way cannot fail to appreciate the contribution of MODELS, which is also fully acknowledged in the Programme Director's own account. We pick up on this feature of the programme in the Evaluation section under Programme and Project Management.

5.2 New directions - Hybrids and Clumps Domains

The new thinking behind the programme is embodied in practical form in the development of two new domains. They build on the understandings, such as:

a) that digitisation and copyright clearance are too expensive for purely networked electronic responses to be provided at this point; and

b) that tools for resource discovery need to be linked to those of identifying locations.

‘The hybrid library was designed to bring a range of technologies from different sources together in the context of a working library, and also to begin to explore integrated systems and services in both the electronic and print environments. The hybrid library should integrate access’ (Chris Rusbridge)

‘...using different technologies from the digital library world, and across different media. [The] name hybrid library is intended to reflect the transitional state of the library, which today can neither be fully print nor fully digital. As we have seen, in so many cases the results of adding technology piece-meal are unsatisfactory. The hybrid library ties to use the technologies available to bring things together into a library reflecting the best of both worlds.’(ibid.)

‘The clumps area was derived from attempting to solve the problem of access to (mostly print) scholarly resources anywhere in the UK. The need for this was brought home by the document delivery projects, which provided information about resources without identifying their locations. (JISC/eLib) felt that extending COPAC to include all 200 institutions would be expensive and was potentially unnecessary given the potential of z39.50. Further virtual union catalogues would potentially have access to more current data and more accurate status information than would be possible with a physical union catalogue. The idea was boosted by plans to enhance co-operation between libraries arising from the Anderson Report, now coming to fruition through the Research Libraries Strategy of the HEFCs.’(ibid.)

Hybrid Libraries annual reports available to us included those from: BUILDER; HeadLine; HYLIFE; Agora, and Malibu.

Clumps annual reports available to us included those from: CAIRNS; Music Libraries Online; RIDING, and M25 Link.

5.3 Hybrid Libraries

There are five Hybrid Libraries Projects. BUILDER seeks to develop a working model of the hybrid library within both a teaching and research context, seamlessly integrating access to a wide range of printed and electronic information sources, local and remote, using a Web-based interface and in a way which will be universally applicable. The Web server now supports JavaScript, PerlScript and VBScript. The site uses ‘user session tracking' and a 'front door' for authentication and user profiling. The Website front-end has already been revised and updated during May 1998 to make it more user friendly and externally-contracted formative evaluation has been built into project planning throughout the life of the project. HeadLine sets out to develop and implement a working model for a user-centred hybrid library within the disciplines of economics, finance, business and management at three partner institutions. The model is, however, intended to be scalable to other subject areas and/or larger consortia. HYLIFE addresses the integration of the delivery of large-scale, print and electronic service behind an electronic interface. The focus is on users and on organisational, social and educational issues rather than technology, exploring the interface demands for different user groups, including: practitioners; franchise college interface; research interface, remote users, teachers; and distributed users through a widely distributed consortium. The Agora project seeks to ‘distill the essential components of a hybrid library system...producing a software system and informing the HE library community of the issues of interacting in a distributed electronic resource.’ Malibu sets out to develop three prototype hybrid libraries for the humanities at partner institutions ‘to include and integrate many new services; and to develop organisational and management models for the hybrid library, based on the prototypes and tested at other test-bed institutions.’ Dissemination forms an important strand in all these projects.

5.3.1. Cultural issues

Already, however, some of the problems faced by Phase 1 and 2 projects are resurfacing. For example BUILDER reports:

‘There has been a specific initial problem in securing the co-operation of members of academic staff in collecting material for the Electronic Short Loan collection. Initially it was planned to use representatives from departments to collect reading lists from their academic colleagues on behalf of the Project. However, some of the members of staff concerned did not collect the material within the agreed time scales. It has now been decided to approach all members of staff in the relevant departments directly in order to request materials. It is hoped, however that there is enough 'slack' in the Project time scales to ensure that this delay will not impact on the ESL key milestones.’

HYLIFE explicitly sets out to provide an exploration of these issues involving users iteratively through the development phases. However, even here a risk analysis undertaken by the Project ‘did not fully anticipate the difficulty in gaining access to users for the initial user needs analysis, nor the time that this activity would take’ so that increased effort with parallel promotion of the service will need to be developed.

5.3.2 Digitisation and Copyright

Both in-house and contracted out potentials will be explored for digitisation and copyright clearance in BUILDER. BUILDER has identified various data providers with whom they would like to achieve greater interconnectivity. HYLIFE has developed a few general principles of hybrid library interface design across its consortium:

a) users want an interface to hybrid information resources, not a HYLIFE project add-on. Therefore heavy use of the project logo in the interface is to be avoided, and institutional branding is to be preferred.

b) links from the interface to apparent services, which are in fact information about services are to be avoided. Having received supporting information about the target services within the hybrid interface, users will become frustrated and disillusioned if they then encounter too many introductory screens proper to the target service.

c) The description of the hybrid services within the interface must be approached with care. The advantages of an 'eLib sponsored gateway' or even a search engine may be readily apparent to a practitioner, but an undergraduate may not understand what this means, and consequently be deterred from using the service.

5.3.3 Future directions and developments

BUILDER aims to produce an exemplar of how significant parts of the hybrid library could develop. Some of the products developed by BUILDER will be full-scale services relatively quickly. Others will be produced in the first instance as prototypes for particular user groups and then if appropriate scaled-up. Issues of scaling and portability will be considered throughout the Project.

Malibu reports that:

‘What has become very evident in the first six months is the impact of innovations taking place outside the projects, and the problems caused by the variance in rapidity of changes in different areas. Some of these changes are institutional developments and can provide useful opportunities to MALIBU,... or can create an uncertain situation for staff, such as the merger at Kings College London. ...The resource/information providers market is changing rapidly, resulting in changing conditions both in legal conditions and technical functionality while at the same time innovation in the academic end-user environment and work-practices is slow. The challenge will be to find a balance between all of these innovation processes.’

5.4 Clump Projects

CAIRNS is setting up a SCURL (Scottish Confederation of University and Research Libraries) clump, also configurable as a number of more specialised sub-clumps, comprising servers at 16 Scottish sites. These will be generated dynamically to allow searching across geographical, material type or subject-based groupings. Establishment of the CAIRNS clump is dependent on the installation of z39.50-compliant Scottish catalogues or information services based at the participating sites. Six of the 14 HEI consortium members have so far installed. CAIRNS aims to establish the issues involved in establishing a comprehensive cross-catalogue service in Scotland. Music Libraries Online aims to pilot the application of z39.50 to the networking of library catalogues into subject clumps, creating a gateway to a virtual union catalogue for music materials of all kinds in British libraries and addressing bibliographical, technical and management issues.

More limited data on RIDING and the M25 Link Project were provided in their reports since these were still in the first few months of operation when reports were due. These are effectively geographical clump projects. M25 Link aims to provide a useful and usable service for large scale resource discovery in London.

5.5 Cross-Phase 3 issues

Agora reports ‘eLib does not provide enough money to do the job properly. The project intentionally appointed in June to conserve resources. This has put back some of the planning activities until all the team were assembled. The lesson learnt is that it is better to get staff on earlier to build a shared vision even if they have a limited role while a project prototype is being constructed.’

Like Agora in the Hybrid Libraries domain, Music Libraries Online reports that the project is underfunded and that ‘more specific guidance on the appropriate level of staffing for a project of this nature would have helped to avert a situation where the project has to depend heavily on the voluntary commitment of a few people.’ They have noted problems of burnout referred to in the 1996 Synthesis of Annual reports which has resonances for them.

Since these are important issues, we should record the Programme Director’s response to an earlier draft of this paper pointing out that projects should not bid if they do not think they can deliver within the resource constraints. The MLO budget was increased from the original bid.

Learning from the process of implementation at MLO includes: they intended to register a new domain MusicOnline.ac.uk for the project which would reflect the consortial nature of the project and provide for continuity if eventually supported by multiple servers physically distributed across the UK. Permission for registering this domain was sought and granted by JANET. However they report that,

‘UCE were not prepared to allow us to use this domain to name a server physically located on their network, insisting that all machines on their network had to be known by a name on the uce.ac.uk domain. UCE is not alone in insisting on stringent security policies as regards to their networks. Such policies, however, tend to restrict the use of the network to aid communication and dissemination...a new location may be considered as part of the exit strategy.’

Music Libraries Online identified a need for early joint liaison between Phase 3 projects and systems suppliers in order to explore common solutions to technical issues, to eliminate duplication of effort and to establish Phase 3 customers' needs on the suppliers' R & D agenda. "After subsequent meetings it became apparent that the needs of the clump projects were sufficiently distinctive to merit continued joint meetings....MLO remains committed to the principle of a collective approach, but has learnt through experience that, in voluntary arrangements such as this, clear deadlines in relation to individual project milestones need to be agreed.

HeadLine concurs on the advantages of a shared approach, commenting that ‘some benefit has been gained from liaison and collaboration with other eLib Phase 3 Hybrid Libraries projects, particularly in the coordination of planned dissemination events.’ The project also identifies complications associated with the volume of documentation required by eLib during the project start-up, so that ‘complying with these requirements has caused some delay in the finalisation of a detailed workplan, and therefore in the commencement of substantive work according to that plan.’

5.6 Conclusions

When we draw attention to some of the same phenomena emerging in this third phase of the programme which dogged its earlier phases, we do not mean to belittle the programme's real achievements. We see both the outputs of the first two phases and those of the third within a framework of action research where even the negative outcomes provide important learning. There are also many tangible achievements which individual projects can point to in Phases 2 and 3. Hybrids and Clumps represent a new way forward for explorations of electronic library provision in the UK whether the new Clumps domain strategy is successful in plugging gaps in subject coverage remains to be seen. These new projects and domains will hopefully fulfil some of the necessary functions of transitional institutional forms while new institutional realities are negotiated. These are addressed in our final section on Institutional and Cultural Change. Before then we pick up on some of the Programme and Project Management issues which have pertinence to projects in all stages of the programme.

6. Generic Issues in Programmatic Progress

6.1 Introduction

In this section we review a number of process issues which continue to affect programme performance significantly in the third year of the programme. These include:

• staffing

• management of consortia

• funding

• business plans and exit strategies

• evaluation

6.2 Staffing

The enthusiasm and commitment of staff at all levels within this programme has created its many achievements; in general project annuals reports suggest that the programme successfully mobilised skills in a fruitful way. Our discussion about staffing should be seen within that context. However there were a number of problems which might usefully be highlighted as learning from this programme.

For dedicated programme and project staff the labour market is relatively restricted: Involvement in such projects does develop skills but individuals also need to be at a point in their career where they can work in a time-limited programme of this kind. At the programme level, this can mean actors being seconded from full-time permanent positions. At the project level it implies a somewhat mobile labour force, likely to move as other opportunities present themselves, especially if these offer longer-term security and advancement. At every stage we have noted the impact on projects of short-term contracts and we had every expectation that the final year of Phase 1 and 2 projects would be the worst, since towards the end of the lifecycle of the phases staff would be likely to look elsewhere. This was largely, but not entirely borne out:

- Those projects which have continuing life, may continue to offer employment to staff.

- Some projects have anticipated staff leaving and have built in flexibility and opportune deployment of different kinds of resources.

A few projects reported major disruption in Year Three and a number reported more minor problems; others reflect on how earlier staff losses resulted in disruption of schedules.

‘Staff retention has been an ongoing problem. We had expected that it might be but we did not expect to lose staff during the first year of the project or so regularly throughout the three years. In the last reporting period alone we have lost four staff. The problem is partly because staff are themselves highly visible and have the insecurity of being on short term contracts, and partly because after a year with the project they have gained extremely marketable skills.’ (Netskills)

Non-dedicated project staff, especially those in management roles display some signs of over-commitment and burnout. The degree to which the project objectives map onto their full-time positions and objectives may determine how they respond to project demands. We would, however, be surprised if there were not some distancing of project managers in certain cases and at particular times. Since these actors often formed the chief institutional links with the HEI consortium members this should be given continuing attention in further phases of the programme.

We note, with some disappointment, the absence of annual reports from a number of key projects. In some cases we tried to chase down these projects. Commercial sensitivity may account for some projects failing to report, but it would be regrettable if staff disruptions should result in a loss to the programme of the learning from these projects.

Bottlenecks in terms of skills available to the programme as a whole may also be seen in terms of other consortium members and suppliers of services. These might be characterised as external actors whose role becomes key to a number of different outputs. Providing timely phased inputs from other players is an area where the programme could create more leverage by providing for contracts for retaining the services of such suppliers. Fretwell-Downing is most often the named in this regard but there may be others. We ourselves at the Tavistock are not immune from the types of pressures which result in the kind of optimism described here.

‘It is fair to say that Fretwell-Downing have acknowledged that their enthusiasm for winning work sometimes outreaches their resources’.

but also..

’...this is too terse and one-sided to be fair to Fretwell-Downing who have been an honourable partner’.

Critical suppliers of services who are collaborating across different programme areas might be seen as needing different kinds of contracts which explicitly acknowledged the programme as well as the project role they play. Niche players like Fretwell-Downing clearly make an outstanding contribution to the programme which outweighs the difficulties in working with them. The task is therefore how best to facilitate that contribution and maintain their continued involvement.

6.3 Programme and project management of consortia

The weakness of the consortium approach overall is the lack of institutional ownership which can allow projects to drift as a result. There has been ample evidence across the projects over the first and second years of the time-consuming nature of managing consortia and Year Three reports continue to reinforce this message.

Seren, for example, reports:

‘The Project resulted from the forced merger of two proposals without any clear direction from eLib as to what they expected from such a merger. As a result the need to give equal status to the two original proposers, the University of Wales, Cardiff, and NEWI, led to the establishment of a distributed development team in a country where north-south communication is difficult and time taking. ... The fact that the site chosen (by eLib) to manage the Project had been only peripherally involved in only one of the original proposals was unhelpful.’

Scope reports ‘insufficient liaison between administrative and technical teams, resulting in miscommunication, confusion of roles and responsibilities and further delays’ particularly due to changes in Project Manager.’ And they plead, ‘please build in realistic slippage to ensure pilot testing happens before service is due online.’

Redoubled programme action in these areas is desirable: building on the project management guidelines and workshops for project managers, perhaps by providing more consultation to consortia in conflict etc. and more early intervention to head off problems.

This raises the question as to what extent programme managers can realistically take responsibility for the projects, given the many calls on their time. This question is especially relevant in relation to the realistic timing and resourcing of projects.

Hopefully the final summative evaluation will seek to answer the questions about which kinds of support functioned well in the overall programme and which less well. What can be clearly said now is that investment in programme management resources is vital and does pay off.

6.4 Scale of project funding

Project reports from the third year again raise questions about the judgements about the level of funding to projects which have been around since the beginning of the programme. At the programme’s outset these were raised mainly in relation to software development, which several projects complained did not fit the kinds of timescales this programme was operating within. These concerns are especially important in our view since they are already surfacing in the Phase 3 projects as well as providing material for reflection in Phase 1 and 2 projects.

In terms of the focus of this paper which is programmatic learning, this may be impaired by projects which fail to report or by projects being under-resourced. A number of projects have drawn attention to this, but we offer an example which highlights a continuing problem in the programme:

Seren felt they were under-resourced as a result of the bidding process:

‘Competitive bidding for funds discourages proposers from inflating costs unrealistically but it can have the opposite effect - especially when those constructing the bid do not have long experience of projects involving software development. An added incentive to keeping costs down in the case of this proposal was the need to show a significant proportion of funding from institutional contributions; in the case of the relatively small Welsh institutions, at a time of ‘consolidation’ and ‘efficiency savings’ this made it imperative to keep the total as low as possible. Unless the Funding Councils wish to discourage bids from small institutions and ones which do not have a long history of project work, a process in which the funding could be discussed and negotiated, once the worth of the proposal was established, would be more likely to guarantee successful outcomes.’

Both Agora and Music Libraries Online are among Phase 3 projects which feel that they are under-resourced to deliver adequately to the objectives set for them. "HEFC is always quick to knock down funding budgets but rarely increases budgets in projects which are clearly under budgeted, even at the initial proposal stage" was one comment from Seren which resonates here.

While eLib seeks to pursue a strategy of penetration across HEIs, under resourcing of particular projects remains a distinct danger. A balance needs to be struck between what eLib wants to explore and investigate through properly funded projects on the one hand, and on the other hand the need to reach as many HEIs as possible through lower-funded projects. If projects are underfunded then their objectives need to be curtailed from the outset. While we take the point made by the Programme Director (quoted in 5.5) about the responsibilities of projects themselves, it is in the interests of the programme as a whole to shoulder some of the responsibility for under-resourcing as it emerges.

6.5 Business planning and exit strategies

The consortia approach did not always lend itself to business planning and planning exit strategies. While most projects made commitments to implement exit strategies at the outset, in general progress was not sufficient for products and services to be launched in the way envisaged. Our impression from the Year Three reports is that exiting pressures drove project staff to invest in particular institutions, often those in which they were physically based.

Although many projects developed valued new services and products during the life of the first three years of the programme, these were rarely viable in commercial terms. There are also some important domain or area differences. For example, it would appear that many eJournal projects can survive in some attenuated form from a more restricted institutional and funding base than that enjoyed over the life of the programme. On the other hand EDD projects concerned with software development like EDDIS and Seren, who are still developing their product during Year Three, see less clearly into the future. EDDIS has been more successful in exiting, according to the reports we have, winning on-funding within a plan for ultimate exploitation. As noted in our Year One report, however, Seren’s lack of institutional embeddedness has made it extremely vulnerable when looking for continued support.

Access to Networked Resources projects within the programme were also facing an uncertain future at the time they drafted their annual reports for Year Three. In this case the area of uncertainty surrounded the overall framework for resource discovery being developed at the strategic level, into which they would need to tailor their future development.

In many cases there were simply not the skills and experience at the project management level to launch these new products and services into commercial exploitation, even when there was some previous track record of success, as in EDDIS: ‘On the business side of things we embarked on EDDIS with experience based on the success of BIDS services and Lancaster’s ILL product. In retrospect we might consider that both of these are special cases and the business challenge facing VDX/EDDIS is different. Certainly there is a big gap between having a product and having a business, especially when there is no pre-existing business, income or arrangement to underpin its launch.’

6.6 Evaluation

Evaluation activities have been a particular focus for projects reporting on their final year of operation or funding. At the same time, it is clear that what evaluation has meant varies widely; for some projects it has been comprehensively undertaken by professional external actors. It is common for continuous formative evaluation to be emphasised over summative evaluation in the early stages of project life, and the need to build this into project planning was certainly emphasised by us in our first year’s synthesis. Appraisal of the overall vision informing project plans and activities is an element of evaluation which is clear from evaluation guidelines but which may have been lost to view as projects got underway.

From the start we have tried to distinguish between evaluation related to usability and evaluation related to usefulness. The Evaluation Plan Checklist on page 30 of the Evaluation Guidelines asks:

‘Will the evaluation and assessment activities identify cost-effective solutions to real user needs in terms of functional effectiveness, institutional or disciplinary relevance and potential for growth, transfer or dissemination?’ (emphasis added)

However, in the initial stages of projects it was perhaps inevitable that usability aspects should have received a greater emphasis as demonstrator products and services experimented with different roles and niches. Often those users who were prepared to give time to help test and evaluate were those committed to the technology, for whom the payoffs between usability (or costs of use) and those of usefulness (or benefit) are different from those enjoyed by novices. Long-term usefulness has also been difficult to assess in many cases until other issues have been resolved: e.g. critical mass of desired text digitised and/or cleared for copyright; or e.g. institutional issues of access to technology and cost recovery have been settled.

Our first Synthesis of Annual Reports which covered many project’s start-up activities during the course of 1996 comments that some projects were disappointed that their evaluation plans were curtailed at this point. At the same time there was already some evidence of a tempering of the grand ideas which had informed original project visions. One project reported itself as aware it would produce new knowledge and understandings rather than a specific service or product.

As we look over the fairly extensive evaluation activities which have been undertaken, for a variety of reasons projects have been more successful in generating what is sometimes termed ‘single-loop learning’, (as described in evaluation examples in an earlier section in Access to Networked Resources), learning about how to do what they were doing better. We can contrast this type of learning with that embodied in the decision to develop hybrids and clumps in the section which follows it, an example of ‘double-loop learning’ - learning capable of generating choices about what to do rather than going on doing the same kinds of things.

We noted in Year One that projects were reporting some discomfort that evaluation plans had been curtailed in a way which would make it more difficult for them to answer their double loop learning questions: is it worth doing this kind of thing, etc. On the whole projects seem to have been more successful in answering how questions for users - how will they use it, get to it, etc.? rather than why would they want to use it?

However, in the Year Three reports we noted with some satisfaction that a number of projects took up our invitation to reflect more broadly on their experience of the programme if it was the final report that they were producing. A number of projects, including some whose previous accounts of their projects had been somewhat impenetrable, responded to the opportunity for reflection provided with candour about the shortcomings of their approach and resources and the lack of opportunity they had had to test out doubts earlier on.

7. Training and Awareness - Engagement and Cultural Change

7.1 Training and Awareness

Follett argued strongly for the need for a full national training programme for librarians. The mechanism suggested was (implicitly) a resource based programme concerned with developing training materials to support network use by librarians and, in turn, library users, plus pilot projects aimed at increasing shared use of training resources. In addition, the programme would provide in-service courses directly, identifying training and continuing education needs of librarians and, more generally, promoting the use of network resources.

Three projects’ annual reports were available for the third year synthesis: Netlinks, Edulib and Netskills. Overall in the programme, the domain area comprised a diverse range of activities which suffered from lack of linkages between them. We have previously remarked that: ‘it may be better in future to arrange such support work by commissioning or call to tender for a specific brief rather than by calling for proposals, and appropriate targeting of groups to attend the courses, and continuous formative evaluation during course delivery will be important if value for money is to be achieved from the current projects.’

In our synthesis of Year Two reports we commented that ‘In two cases, projects had come to realise the limitations of an individual-centred approach when seen in the broader context of the institutional changes needed to support a wholesale shift to a networked environment’ Projects commented: ‘The strategy we adopted to target individuals in library/information services to undertake co-ordinator and change agent roles in relation to NLS development in their institutions has met with limited success. It is clear that without significant support at the level of strategic planning and the time to take on such roles, it may well be unrealistic in many institutional contexts to expect these staff to become proactive. Nor is it necessarily the case that information service staff are the most suitable staff to take on co-ordination of an area of development in which there is such a diversity of stakeholder interests and perspectives.’

Throughout the life of the programme, projects across the domains have reported constrains deriving from the slow pace of cultural change. Staff involved in projects have by definition been enthusiasts for the new technology and lack of interest has come as something of a surprise to them. The cultural changes targeted by the Training and Awareness project is a critical precondition for access of many of the products and services being developed within the programme as a whole.

The lack of institutional embeddedness in the training strategies make evaluation of their progress against the original aims for the area extremely difficult to judge. In this years’ annual reports, Edulib comments:’ We have welcomed the opportunity to be subject to independent evaluation and we have no criticism of those who have been involved with the evaluations. No doubt JISC and others may gain worthwhile information from independent evaluations which will enable them to demonstrate that tax payers’ money has been employed appropriately and effectively. From our point of view, independent evaluation has consumed much energy, time and resources but has not yielded any useful information or data which has informed the implementation of our project or indicated whether or not we are being effective. This is because Edulib is not producing some sort of tangible ‘product’ which can be readily contained and ‘measured’ in some way. Edulib is an educational intervention in the developmental process within higher education. As such it is one element in a complex and somewhat idiosyncratic social process which may, to use the jargon, be illuminated, but which cannot be audited in any more systematic or rigorous manner. The problems associated with conceptualising an innovation such as Edulib for evaluation purposes and then designing and properly resourcing an evaluation programme which would yield valid and reliable data are intellectually and financially overwhelming.’

While we have some sympathy with the views expressed about the demands of evaluation in relation to the learning generated, the work undertaken by these projects must be subject to some kind of evaluative activity without focusing on individual change in which the impact of different social factors are hard to distinguish. The Impel 2 evaluation of Netskills should generate some data through its case studies investigating the social, organisational and cultural issues surrounding the management of an increasingly electronic environment but the results of this analysis were not available to us.

If the domains or areas defined by projects can be described as part of the programme architecture, then Training and Awareness projects address features of the landscape in which they are situated. Success for these projects may not be seen in terms of outcomes for the eLib programme. New skills and approaches nurtured by Netskills for example, are likely to undergo a process of naturalisation during which many other products or services than those evolving through eLib may benefit equally or more. The relocation of the Training and Awareness projects outside the boundaries of the eLib programme reflects the continuous nature of the inputs needed in this area.

7.2 Institutional engagement

Institutions are largely lacking in the picture which most other eLib annual reports presents. They appear as spaces with constraints or opportunities for things to happen in but there is usually no coherent strategy for changing those constraints or expanding those opportunities. Consortium arrangements rarely include senior members of the hierarchy (although we note that BUILDER explicitly aims to take account of these). And yet institutions are key to change, since to exploit the technology effectively requires that a lot of change takes place at different levels simultaneously. Institutional interventions need to be built in at all levels. Institutions not only provide the constraints and opportunities they also mediate the various norms and expectations of different professional groups. The interaction of different professional groups commented on in the annual reports has mainly concerned the interaction between librarians and academics. But those between computing staff and librarians have also been identified as an issue.

Netskills’ third annual report refers back to research they conducted in 1995-6 reporting key issues identified by them at the institutional level:, including: the need for senior managers’ involvement; the diversity of institutions, organisational types, administrative structures, convergence and non-convergence of services requires that a variety of approaches and interventions is necessary; strategic moves at an institutional level towards network-based learning must embody networked learner support; and networked learner support demands collaborative activity between IT services, information support staff and academic staff.

The progress of cultural change would be easier to assess if there were more indicators at the institutional level. Institutional strategies with milestones and objectives would create structures within which cultural change could be resourced and evaluated without such structures. Cultural change can only be assessed in terms of the training outputs rather than in terms of the changes in behaviour which would indicate its impact.

HEIs at the senior managerial level appear to be developing strategic goals in relation to electronic library services. This is an area where JISC might usefully develop complementary activities to the eLib programme as a way forward. Rapid institutional change can be experienced as culture shock. Lack of institutional change during a period of cultural change will tend to slow the pace of change causing inertia and underlining the motivation and commitment it needs to harness. In socio-tehnical change scenarios the best advice is often to "change everything at once". The process of managing change at the level of HEIs is a political one, requiring persuasion, negotiation and compromise. It is not obvious to us from this set of reports that new roles for HEIs are under active consideration by JISC or that the need for this is understood within the programme. The concomitant need for institutional change among partners such as publishers is likely to follow institutionalised changes in the market which may be some time in coming.

7.3 Cultural Change

Researchers at De Monfort University have offered the following practical definition of cultural change, building on previous Tavistock work:

‘Cultural change is composed of

• lasting structural and social changes (within an organisation or set of linked organisations), PLUS

• lasting changes to the shared ways of thinking, beliefs, values, procedures and relationships of the stakeholders within that grouping.’

When we consider this definition it becomes clear that establishing cultural change in the HE library community does not simply concerning training and awareness raising activities but must also address the substantive types of material change envisaged in the hybrid libraries and clump projects. In a sense we should consider these Phase III projects to be ‘path-finding’ or ‘pilot’ projects in organisational change and cultural change.

Such an approach does leave us with some important questions which should be examined in 1999 as to how successfully hybrid library and clump projects are accomplishing both material developments to organisational systems and work processes and whether these material changes are being adequately supported by simultaneous staff development measures.

8. Conclusion

While overall judgements on the outputs and impact of eLib, and, indeed on its value and worth, will await the independent summative evaluation planned for 1999, it is clear, from this review of annual reports, that eLib has not only fulfilled its Follett brief but has already critically affected the ways in which the community understands the opportunities for the development of HE library services.

Further it is arguable that the single major output of Phases I and II of eLib is already observable as it is the conceptualisation and launch of the Phase III hybrid library and clump projects which will be both demonstrations of the potential of the new library technologies developed under eLib when integrated in practice and pathfinding projects in organisational and cultural change.

Beyond this, 1999 will see the ‘scaling up’ of some of the more important elements of the eLib programme in various guises including a major initiative in On Demand Publishing through project Heron, a new strategy for resource discovery with the launch of the Resource Discovery Network Centre, and a Digitisation centre. Other elements of eLib, particularly the future development of electronic journals and extensions to offered Electronic Document Delivery services will probably proceed at a lesser pace but it is reasonable to expect that further developments which will emerge in these areas as markets mature and academic cultures evolve will be able to build upon the extensive ground clearing achieved by eLib.