Evaluation of the Electronic Libraries Programme

Policy Mapping Study - The set-up, operation and content of the Electronic Libraries Programme

Final 2

August 1996

Revised October 1996

The Tavistock Institute

Evaluation Development and Review Unit

30 Tabernacle Street

London EC2A 4DD

Tel: 0171 417 0407

Fax: 0171 417 0567

email: edru@tavinstitute.org

Contents

1. Summary

Introduction

This study maps actual and planned activities in the Electronic Libraries Programme (eLib) against programme objectives. It is intended to document the programme’s content and organisation; indicate possible directions for future activities; and provide a baseline for future reference and evaluation.

This study should be read in the context of JISC’s(1) own recent policy studies and statements.

The study has been carried out on the basis of data from extensive interviewing and documentary review within and beyond the programme. The study itself has taken the form of an analytical exercise in four steps:

1. An inventory of programme policy objectives,
2. A mapping of these objectives onto actual programme activities (projects),
3. An analysis of the likely contributions of these activities to achieving policy objectives, and
4. An identification of limitations and scope for future action in fulfilling programme objectives.

This document is divided into four main sections: Programme Context; Programme Objectives; Programme Content; and Programme Future.

As programme objectives have been refined and elaborated over time in the light of changing experience, there is an extent to which the programme content always, with hindsight, falls short of the evolving objectives which FIGIT is setting itself. For this reason, where limitations and lacunae have been identified in this study they should be read in the first instance as policy learning rather than as policy or operational failure.

Programme Context

eLib was set up in 1994 under the auspices of the Higher Education Funding Councils’ (HEFC) Joint Information Systems Committee to implement many of the information technology recommendations of the Libraries Review Group chaired by Sir Brian Follett.

The programme runs from 1994-1998, guided by FIGIT(2) and a full time Programme Director, with funding of £15m, and consists of sixty projects and a number of additional supporting studies.

eLib has a number of distinct organisational characteristics, notably;

• an emphasis on learning, mobilisation and catalytic effects,
• a management strategy which is both directive and consultative, and
• a conception of electronic information which goes beyond library provison to include elements of scholarly communication, teaching and learning.

Programme Objectives

The specific areas needing action identified by Follett included copyright, standards, electronic document and articule delivery, electronic storage of books and journals (digitisation), electronic journals, on demand publishing and the electronic book, awareness and training, navigational tools (access to network resources), networking, databases and datasets and catalogues, retrospective conversion of catalogues, library management systems, and some other general tasks.

The programme initially developed following the Follett report, was essentiallly a practical specification, modelled entirely within the framework set out by Follett, with a few minor shifts in emphasis, notably in the areas of copyright and standards, and a few omissions and additions. A number of elaborations were also made, particularly as concerns consortia composition, electronic journals, imaging, short loans and digitisation.

In the light of experiance, FIGIT has since 1995 evolved a series of proactive norms for the management of the programme which emphasise project management, evaluation, communication and exit strategies. More generally, the overall programme objective was settled as “to stimulate change within the information chain” and “to engage the Higher Education community in developing and shaping the implementation of the electronic library” particularly as regards developing operational systems and seeking cultural change.

Programme Content

eLib, as a programme, is divided into the following areas of activity - Electronic Journals; Pre-Prints; Quality Assurance; Access to Network Resources; Digitisation; On Demand Publishing; Electronic Short Loan Collections; Electronic Document Delivery; Supporting Studies; and Training and Awareness. In the main body of the text, for each area, we lay out, in the form of a summary table, a discription of area projects indicating objectives, actual content and (realistically) expected results. In the case of some areas we also offer an opinion on the area project content.

There are no significant areas mentioned or implied by the Follett report, or indeed in the wider literature, uncovered by some HEFC library related initiative. A detailed mapping of this by area, and a summary, by area, of eLib projects is contained in the main body of this text.

Project partnerships/consortia teams consist mainly of librarians or academics, and to a lesser extent senior university managers. In our view, this consortium structure supports networking across institutions, but it also appears that some projects will be limited in their ability to exercise leverage upon their own institutions.

Programme Future

We make two main recommendations regarding programme future, one relates to integration, the other to cultural change:

Promoting integrated use of services and achiving critical mass in service provision implies actions and investment, to some extent simultaneously, at the level of the individual HEI(3), individual disciplines, and at the ‘centre’ in the form of JISC services.

Cultural change initiatives, if they are to be successful, must go hand in hand with structual and material change. Individuals’ and groups’ behaviour and orientations will only be changed through awareness raising and training where their work contexts are also changed in a way that makes new knowledge and skills relevant.

A number of other specific recommendations are to be found in the body of the text and relate to Teaching and Learning; Intranets; Institutional Change Projects, Multi-media Use and Disciplinary Environments; Information Resource Discovery and Management; Training and Awareness; Evaluation, and the Promotion of Good Development Practice.

2. Introduction

Study Content

The policy objectives of the Electronic Libraries Programme (eLib) have evolved as the programme has developed. There is now a broad consensus as to programme objectives and a need to evaluate to what extent these objectives have been successfully translated into actual programme activities. This document presents a logical analysis or mapping of programme activities in terms of programme objectives. This exercise is the first activity of the overall summative evaluation of the programme. This policy mapping study is intended:

• to clarify the strengths and weaknesses of the overall programme in terms of content and organisation which are now becoming apparent,
• indicate the type of future activities which are desirable, particularly in terms of future CEI(4) activities, and
• provide a baseline against which programme activities can be related over time to wider policy and institutional goals.

Study Context

This policy mapping study has to be seen within the broader context of JISCís(5) own recent policy studies and statements. This study accepts the general thrust of that policy and its components of network development; electronic information; applications; and awareness, liaison and training. The division between services and development within the remit of a single electronic information committee also seems an appropriate improvement in management coherence. We, also in keeping with FIGIT(6) thinking, see the importance of integrated pilots, but here we find the limits of the JISC remit, that is, the lack of remit to promote institutional innovations. We need to consider then not just whether eLib has fulfilled its remit (which it has), but what is the relationship between eLib and the rest of JISC and between JISC and the rest of HEFC and the individual HEIs(7)? This is linked, as we shall see, to the challenge of promoting cultural change.

Study Design

The study has been carried out on the basis of data collected in the course of the extensive set of interviews and documentary review undertaken in Phase One of the eLib evaluation (development of an evaluation framework(8)). This data has been augmented by additional supplementary interviews with Chris Rusbridge(9), Alice Colban(10) and Malcolm Read(11), review of more recent policy papers(12) and documentation on projects newly funded in 1996,(13) and a review of some other related initiatives, most notably in the areas of teaching and learning(14) and archives(15). The study has been also informed by the extensive, detailed discussions with projects that have taken place in the context of the individual project evaluation 'clinics' around the recent series of evaluation workshops for elib projects. The study itself has taken the form of an analytical exercise in four steps:

1. An inventory of programme policy objectives,
2. A mapping of these objectives onto actual programme activities (projects),
3. An analysis of the likely contributions of these activities to achieving policy objectives, and
4. An identification of limitations and scope for future action in fulfilling programme objectives.

This document provides a synthetic statement of the study findings. The analysis was further refined on the basis of comments from the members of the FIGIT Evaluation Sub-Committee(16) on the formal descriptions of programme objectives, content and future contained in an earlier draft.

This Document

This document is divided into four main sections:

Section 2: Programme Context rehearses the origins of eLib and the programme's place within the broader JISC structure.

Section 3: Programme Objectives contains a formal resume of the programmes objectives, formally and as they have come to be refined and elaborated in practice.

Section 4: Programme Content looks at the actual content of the programme areas and overall and examines the degree to which they cohere with the programme objectives.

Section 5. Programme Future examines the present scope for further action in accordance with the programme objectives.

As programme objectives have been refined and elaborated over time in the light of changing experience, there is an extent to which the programme content always, with hindsight, falls short of the evolving objectives which FIGIT is setting itself. For this reason, where limitations and lacunae have been identified in this study they should be read in the first instance as policy learning rather than as policy or operational failure.

3. Programme Context

The Electronic Libraries Programme - Brief History

From its inception in May 1992, the Higher Education Funding Council for England identified library provision and facilities as a key area of concern. Following the first meeting of the HEFCE, the Libraries Review Group was commissioned through a joint initiative of the UK Funding Councils (HEFCE, HEFCW, SHEFC and DENI). The Review Group was chaired by Sir Brian Follett (Vice-Chancellor, University of Warwick) and met from October 1992 to October 1993. Its report, the Follett Report, was published in December 1993.

Chapter 7 of the report focused specifically upon IT and resulted in 18 recommendations out of the 46 in total (see Section 3: Programme Objectives). Following Follett's advice, these 18 recommendations were subsequently addressed, as Follett suggested, by a programme ìunder the auspices of the Funding Councils' Joint Information Systems Committee [running] for a finite period and ... guided by an expert advisory group and supported by a programme co-ordinatorî.

The Funding Councils agreed to allocate an initial £4.75 million for 1994/95 (with the expectation that the programme would be funded for a further two years) and to pass on the responsibility for its implementation to JISC. Eventually the programme was to have agreed a budget of £15 million over the period 1994-1998. Following the February 1994 meeting of JISC, FIGIT was set up in order to implement the IT recommendations of Follett, and was chaired by Lynne Brindley (Librarian and Director of Information Services at the LSE). A framework document (Circular 4/94) was issued in August 1994 which described FIGIT's approach to the Programme and invited initial expressions of interest for projects relating to the specification of programme strands with a closing date of October 1994. Essentially the programme strands followed the issues identified within Chapter 7 of the Follett Report (On Demand Publishing, Electronic Journals and so on(17)). In January 1995 Chris Rusbridge formally took up post as Programme Director, just before the start of the first tranche of projects under the programme. Over the first phase of the programme, 41 projects were funded, of which 40 remain.(18). In addition a number of supporting studies (looking at additional issues identified by the Review Group) were commissioned or expressions of interest invited. A second call for proposals was made later that year for projects in areas under-represented or uncalled for in the first round. A further 19 projects were commissioned to begin in 1996.

At the same time, efforts were made to attempt to maximise project effects and success by a number of additional measures, most notably, the establishment of steering mechanisms for the projects, the commissioning of workshops on project management, the establishment of an open electronic communication mechanism for the programme based on Web sites and a programme mail list, the holding of various open and closed meetings involving project participants and members of the wider community, and the development of a programme evaluation framework.

Background to the Follett Review and the Policy Context of the eLib Programme

When commissioned, expansion of HE appeared set to continue and Follett's initial remit was to examine library and related provision in the light of the growth of undergraduate student numbers, changes in demographic profile of students and new styles of course provision and learning approaches. Libraries were seen to be facing a crisis of spiralling costs - particularly of periodicals and specialist books - and pressure on space. It was evident to the Review Group that implementation of its recommendations would be unlikely to result in cost savings. However, the allocation of additional resources would be necessary in order to provide a more cost effective services in the long term - particularly through deploying IT. Thus, the Group focused upon a broad strategy that would contain costs while improving library provision, capitalising on developments in information technology, and streamlining both the management and integration of information services within HEIs.

The terms of reference for FIGIT (Circular 2/94) were to ìensure the cost effective implementation of the IT recommendations of the [...Follett] Reviewî. These recommendations dealing with IT were part of a wider approach which sought to encourage more efficient and forward looking exploitation of information service developments through an integrated strategy for library development. This strategy included greater organisational coherence within universities, more integrated management and operational structures, and greater sharing and co-ordination across institutions with respect to library provision. What became the eLib programme, then, was one part of a much wider set of library related objectives which were logically connected in policy terms. This is acknowledged in the terms of reference of FIGIT (Circular 2/94) in its objective to ìensure the IT developments are linked and integrated with the wider aims and implementation of the Joint Funding Council's Libraries Reviewî and in the objective of close liaison with JISC on developments and policy issues arising from the programme.

FIGIT's Strategy for Organising the eLib Programme

FIGIT's strategy for organising the eLib programme had a number of distinct characteristics:

1. On the basis of their assessment of the current state of development of telematics usage in HE, FIGIT chose to emphasise learning, mobilisation and catalytic effects by drawing a large number of organisations and range of actors into the programme through seeking to fund a wide range of relatively small scale collaborative projects. (This approach was in pointed contrast to the focus of the US Digital Libraries Programme on a small number of large projects.)
2. FIGIT’s approach to developing its strategy was both directive and consultative. It was directive in:

• setting out a clear and comprehensive framework for expressions of interest,
• adhering at the outset to Follett’s definition of areas requiring to be addressed, and
• steering the reshaping of first call proposals, and in some cases consortia, towards close fit with programme objectives.

At every stage FIGIT was also engaged in consultation with its HE and library communities. This led to a programme which could be characterised as reflexive, albeit within a firm framework of clear, overall objectives. It included a responsiveness that could be observed in FIGIT’s capacity to learn and evolve programme content in response to project proposals, external developments, and emerging issues(19). A further aspect of FIGIT’s approach in this respect is seen in its use of expert advice and research based on studies of key issues that impact on projects or future programme activity.

3. While the Follett report was concerned with library provision, FIGIT in addressing the issue of the provision of electronic information was (inevitably) led beyond ICT (Information and Communication Technologies) in the library to considerations of uses of ICT directly in scholarly practice and in teaching and learning. This, in effect, resulted particularly in the areas of Electronic Journals and Pre-Prints, in projects which were not addressing new ways of performing old functions but in the development of new modes of action.

4. Programme Objectives

Follett

FIGIT's formal remit was to implement the IT recommendations of the Follett Report and to do this through the funding and management of a coherent libraries initiative. The specific needs identified by Follett are summarised below.

1. Copyright

The report discussed at length the difficulties surrounding copyright in electronic information environments. It argued that dialogue and the creation of mutual trust between the HE sector and publishers were necessary preconditions to the generation of workable mechanisms for electronic information services. It recommended the funding of a pilot project involving a small number of HE institutions and publishers who would produce a model agreement developed from a practical demonstration of an electronic storage, delivery and information management system.

2. Standards

Follett identified the problem of the growth of heterogeneous and often incompatible systems for information access and retrieval. The development of international standards had been proceeding with little input for the UK HEIs and the Review Group recommended that JISC should monitor these standards and also fund a demonstrator project employing interoperable systems across different computer based media, networks and systems.

3. Electronic Document and Article Delivery

Follett's recommendations included the establishment of both subject and metropolitan based services comprising a variety of partnership models such as, for example, publishers and learned societies. This initiative would also have to address the need for supporting hardware and software and be able capitalise upon developments such as Super JANET.

4. Electronic Storage of Books and Journals (Digitisation)

The report argued for a limited number of large scale demonstration projects in this area which would digitise out-of-copyright journals. Follow up work was envisaged for distribution of this material across HE institutions and for the evaluation of out comes.

5. Electronic Journals

Areas 1 - 4 were concerned with bringing library practice up to date with current developments in IT. Electronic Journals, however, were seen as supporting new developments in academic practice and potentially a more radical innovation. The report argued that their status, credibility and their likely threat to conventional publishing should be explored through the funding of a limited number of Electronic Journal development projects.

6. On Demand Publishing and the Electronic Book

More so than EJs, the fully electronic book or academic monograph was regarded as a prospective development. However, the report saw considerable potential in the creation of networked access to and on demand publishing of learning materials or texts - particularly those held in multiple copies and also collated documents that could be flexibly used or customised by readers. It recommended the funding of demonstrator 'on demand' projects mounted at one or more host universities which would also develop appropriate copyright payments systems.

7. Awareness and Training

Follett argued strongly for the need for a full national training programme for librarians. The mechanism suggested was (implicitly) a resource based programme concerned with developing training materials to support network use by librarians and, in turn, library users, plus pilot projects aimed at increasing shared use of training resources. In addition, the programme would provide in-service courses directly, identify training and continuing education needs of librarians and, more generally, promote the use of network resources.

8. Navigational Tools (Access to Network Resources)

Follett noted the proliferation of software tools and net applications that had been developed for network search and retrieval had proceeded largely on a voluntaristic basis and argued for a more systematic approach and recommended funding of a limited number of top level navigation tools to encourage the growth of local subject based tools and information servers.

9. Networking

In order to take full advantage of external network services, HEIs need to have high performance internal networks and access in due course, to SuperJANET. Follett saw this as an important internal matter for HEIs and recommended that institutions should be required to review and detail their network plans and submit these as part of the Information Strategy reports to the Funding Councils (as recommended elsewhere in the report).

10. Databases and Datasets and Catalogues

Follett identified a lack of coherence and integration across faculty, publishers, booksellers in reading list and associated bibliographic materials required by students and the report recommended the funding of demonstrator projects which would develop systems for the integration of such material and demonstrate their advantages.

Also addressed under this area was the possible need for the establishment of a Humanities Datacentre and the report recommended further analysis of these issues should be undertaken by JISC. Follett also recommended that JISC should fund the development of the CURL bibliographic data base as a national OPAC service.

11. Retrospective Conversion of Catalogues

Follett suggested that retroconversion issues required further study and recommended that UKOLN should be commissioned by the funding councils to carry out this.

12. Library Management Systems

The report dealt with a number of issues concerned with the overall management of libraries within, and, in relation to their institutions. The report specifically addressed the need for better library automation systems and greater articulation between library management and new forms of teaching and learning. In particular, the report noted the work of the TLTP initiative and recommended that this and similar programmes should consider the developing library role of IT based teaching and learning services. As regards Management Information Systems (MIS), the report recommended that JISC should fund an exploratory study on MIS specification and development.

13. General Tasks

Two general themes underpinned the reports discussion of IT and Library provision.

Firstly, the need for deploying new developments which could help increase physical storage space and increase efficiency, speed and flexibility of document access and delivery.

Secondly, the need for better integration at all levels:

• at the hardware and software levels to increase interoperability, efficiency and usability.
• at the level of the library information management strategy to integrate library provision within a coherent institutional approach to management, and teaching and learning.
• at the level of the whole institution. Follett argued strongly that meeting the needs of libraries with respect to IT would be best achieved through greater collaboration and sharing of resources across whole institutions.

eLib

The programme initially developed by FIGIT following the Follett report, was essentially a practical specification, modelled entirely within the framework set out by Follett. Areas 3-8 became specific programme lines, and the remaining areas were either addressed directly elsewhere by JISC where suggested by Follett, or were included in an additional programme line as 'Supporting Studies'. Essentially, FIGIT's specification for the programme differed little from the Follett report in terms of its policies. Its approach to the programme offered a clarification and to some degree an extension of Follett but, broadly, it began as practical realisation of a set of prespecified targeted actions.

There were a few minor shifts of emphasis and a few omissions or additions within the programme specification set out by FIGIT in its statement of the programme and invitation for expressions of interest (Circular 4/94). Notably:

• No pilot project exclusively focused on copyright issues was funded. Instead FIGIT chose to tackle copyright management issues broadly across projects and proposers in relevant programme areas were encouraged to involve publishers.
• Similarly, no demonstrator project on Standards was explicitly commissioned and proposers were asked to address this issue within projects where appropriate (e.g. MODELS). FIGIT also produced a Standards document.
• Although expressly invited, in the event, no subject based EDD project was funded.
• Follett's recommendation that JISC should fund a bibliographic integration project was explored through commissioning a study by Aston University. Subsequently, it was felt that market developments were proceeding along this line and required no specific intervention through JISC.
• Similarly, the issue of Library Management Systems was thought best addressed by commercial sponsors.

FIGIT Elaborations

FIGIT also elaborated on Follett in a number of particulars:

a) Consortia Reach

Follett's message of greater sharing and co-operation across institutions is implicit in the organisation of the programme and the nature of the consortia structures. However, it was not explicitly addressed by FIGIT in terms of evaluation criteria for successful bids, although organisational coverage and reach was to be an important factor in some programme negotiations(20).

b) Electronic Journals

Extensions to the ideas embedded in the Follett report were made to Electronic Journals where FIGIT adopted a research, as well as project based approach - for example, including the request for studies on Peer Review and inviting projects dealing with informal communication and pre-prints. Few proposals were received that specifically dealt with these issues in the first round of funding and both areas were specifically targeted in the second call for proposals under the new programme strands as Quality Assurance and Pre-prints.

c) Imaging

Imaging was mentioned only in passing by Follett under EDD recommendations but not initially targeted by Follett as a programme line. FIGIT received a number of imaging based proposals in response to its initial tender for expressions of interest. In the event, the committee decided not to discard these but to commission a scoping study and on the basis of this study funded imaging projects as a new programme line in Phase II of the programme.

d) Short Loans

The Follett report identified the potential benefits of on demand publishing particularly of course materials and texts held in multiple copies. However, it did not specifically recommend electronic access and delivery mechanisms for high demand material in short loan collections. By inserting this area into the second phase of the programme, FIGIT both addressed a gap and implicitly recognised that replacing short loan collections is fundamental to effective on demand services.

e) Digitisation

The initial programme structure included a digitisation strand as advocated by Follett. In part because of the paucity of proposals received in this area and in part because of growing recognition of the need to address aspects of electronic storage and delivery by more centralised means, FIGIT is now about to establish a digitisation centre.

Policy Evolution

FIGIT's relatively limited initial modifications to Follett specification seem to illustrate its earlier construction as a committee prescribed by the Follett remit. Throughout the subsequent changes and elaborations to the programme, the committee has extended its agency, its learning curve and expertise. As a result, the committee has evolved a series of proactive norms for the management of the programme which emphasise project management, evaluation, publicity and exit strategies. The evolution of FIGIT policy is represented in the changing discourse of its statements of aims and objectives. Thus in the first statement set out in Circulars 2/94 and 4/94 (Annex insert) FIGIT's specific objectives (as well as its overall aims as stated earlier) are explicitly tied to the Follett specification and are essentially procedural. Thus in its programme and project development objectives, FIGIT defines its role as integrating IT developments with the wider aims of Follett. These can be summarised as:

• realising a coherent programme of development and piloting that takes account of international developments to ensure full implementation of the recommendations;
• devising various strategies for programme management that include evaluation of project proposals, allocation of funds and financial control mechanisms and the development of operational services by projects and programme promotion;
• an advisory relationship with the JISC and other relevant bodies associated with the Follett Review; and
• consultation with HEIs.

By 1995, there are clear shifts of emphasis if not policy, in FIGIT discussions and written statements of aims and objectives. The eLib programme's rationale and objectives were discussed at the FIGIT January 1995 committee meeting. In this discussion and associated documentation, FIGIT's overall objective is said to be: "to stimulate change within the information chain..." And sub-objectives include:

• making a significant impact on the delivery of information in HE;
• ensuring that HEIs are adequately prepared to take advantage of developments in ICT and as a result secure cost effective library services;
• intervening to secure early development of cost effective electronic library and digital information services and migration of experimental to operational services;
• explore different models of the position of the scholarly community in the ownership of electronic copyright through practical experimentation and dialogue;
• producing practical IT solutions to library problems which can be scaled up;
• promote change in the publication of journals through encouraging new methods of networked scholarly publishing; and
• encouragement of collaboration and partnership.

These statements are general but support notions of proactivity, practicality, experimentation, agency and facilitation as well as being an extension FIGIT's earlier stated 'managed and collaborative approach' (Circular 4/94). In this respect, FIGIT now sees its role as a broad one of change agent in the HE sector through the diffusion of innovatory developments in ICT. This is reflected in the description of the eLib programme contained in the 1996 edition of the programme information pack which states:

“The main aim of the eLib programme, through its projects, is to engage the Higher Education community in developing and shaping the implementation of the electronic library” (Electronic Libraries Programme - Edition 3).

Its chief sub-aims remain:

(a) operational - in terms of developing the organisation and delivery of library services based on electronic systems, and

(b) cultural - in terms of shifts in attitudes and skills among librarians and their user communities.

In this sense, the programme is moving towards the electronic library (though with an expectation that only some components of the concept are relevant and feasible in the foreseeable future).

5. Programme Content

General

As the summary tables (below) show, there is no doubt that the areas identified for action by Follett have been formally covered by the commissioned projects. One could go as far as to say that there are no significant areas mentioned or implied by the Follett report, or indeed in the wider literature, uncovered by some HEFC library related initiative. The following table is offered by way of demonstrating this statement. In terms of policy mapping, then, our attention focuses on how this formal coverage is composed in practice, what could be further improved and what needs to be addressed next. In this and the following section, we comment overall, and by area, on how Follett's main objectives are being achieved, how appropriate is the chosen mechanism, what could be improved and what is the scope for future action.

Summary Table I: Mapping of Objectives by Area and Project Activities

Summary Table I: Mapping of Objectives by Area and Project Activities (continued)

Summary Table I: Mapping of Objectives by Area and Project Activities (continued)

Summary Table II: Areas and Constituent Projects

Summary Table of Areas and Constituent Projects (continued)

* new projects in 1996 largely from the Second Call for proposals

At this point in the discussion of eLib policy and practice, we wish to signal a shift in tone from factual description and related analysis to a more speculative description of the programme based upon the interweaving of empirical evidence and opinion of developments informed by our extensive - albeit still limited - knowledge of the projects. We take it as demonstrated that eLib has, to date, addressed its policy expectations with respect to the Follett Report remit. In what follows then, we offer judgements and suggestions that FIGIT may wish to consider as it moves into the final phases of the programme and any post programme future.

The Projects Overall

Project partnerships/consortia consist mainly of librarians or academics. The degree and extent to which they include senior university managers (for example, in the person of the university librarian who is also on senior university policy and planning committees) is variable. In our view, this consortium structure supports networking across institutions regarding information services, but it also appears that some projects will be limited in their ability to exercise leverage upon their own institutions. While integration at the level of the individual HEI was not an explicit requirement made of projects by FIGIT at the outset, it has emerged as an issue which will require attention in the future:

FIGIT has attempted to achieve coherence at the meta level, but there is limited evidence from our work with projects that, at the micro level of their own institutions, they are seeking to integrate their work within wider Follett objectives or that they are directly feeding into the information strategy development required by HEFC, including changing learning strategies, management strategies and meeting performance indicator targets. (An additional related difficulty here is that project teams are insufficiently balanced - i.e. there is rarely a full mix of expertise (systems and interface development, management, librarian and academic) nor representation of the perspectives of all key groups.) This can be most clearly seen in the weaknesses of most projects exit strategies at this stage (although we understand that this issue will be targeted in the next 12 months). However it is difficult to see how FIGIT could have been expected to pursue the wider vision expressed within the Follett report - particularly as regards integration and co-operation - given the procedural sequestration of the programme into implementing the Chapter 7 recommendations. Further, these weaknesses may in part be simply a function of project size. Many projects are (for good reasons) too small, too brief, and have too few staff to be able to address the full set of change issues implied by their work or to be able to secure deeper institutional commitment.

One additional concern is the question of the proliferation of areas - some of which are technologically and functionally the same under the skin. This possibly simply reflects a funding convenience. However, FIGIT is also concerned to improve co-ordination and programme coherence and presentation and is likely to consider, for example, larger groupings.

The Areas

In each area we lay out, in the form of a summary table, a description of area projects indicating objectives, actual content, and (realistically) expected results based on the best information available to us at the time of writing(22). In some cases we also offer an opinion based on our interpretation of the available evidence and our extensive contacts with the projects(23)

Electronic Journals

Electronic Journals (continued)

Electronic Journals (continued)

Additional Comment: As can be seen from the summary table, the EJ area contains a rich variety of approaches and strategies: from electronic text to full multi-media; from broadcast to communication; from systems development to action research. Not all projects, however, have a disciplinary strategy with a concentration on multi-media, that is, on exploiting EJs as a new medium of scholarly exchange. In many cases such limitations are a function of scale and staffing. What exists is a firm basis for scaling up or launching new initiatives based on the experience of the present projects: perhaps some flagship disciplinary initiatives consisting of full journal environments linked to an ANR gate, digitised resources, and union catalogues. (Social Science and History spring to mind, but also Chemistry and Medicine which raise the issue of piggy backing on established commercial services).

There is considerable naivetÈ in many of the EJ project teams about quite basic questions, such as to who buys (pays for) journals and why, the real costs of the editorial and marketing process, and design and prototyping methodologies. In our opinion, for their efforts to rise above ad hoc, if inspired, experimentation such projects need to be exposed to good practice models in other projects, strong steering, and evaluation plans which require structure in project activities.

Pre-prints

Additional Comment: We fail to see the logic of not including pre-prints within electronic journals (other than the usual, reasonable, arbitrary ones). Certainly, one key issue for pre-prints is their role and integration within the whole journal production process.

Quality Assurance

Access to Network Resources

Access to Network Resources (continued)

Additional Comment: ANR probably represents the most stable and coherent element of the programme. However, there is a need to develop the model. Just as ROADs spans many projects, there may be a need for a 'super project'. (We expect that it may be over optimistic to expect that the current reorganisation of NISS and BUBL will adequately address this). Much of the meta-level R&D issues in resource management and discovery which are being addressed in the US Digital Libraries Programme have no obvious (to us) co-relatives in eLib. This seems particularly true as regards addressing some of the differences in data sets which will be thrown up by linking with electronic archives. (But we make these comments without having had an opportunity to familiarise ourselves with the Dublin core and Warwick framework discussions.)

The subject gateway approach to accessing network resources has been largely accepted by eLib without (it appears) substantial debate on alternatives. While this may indeed be the best way forward, we would have reservations if this model was to be promoted (specifically, funded by JISC) in many more disciplines beyond the lifetime of eLib without a full evaluation of its strengths and limitations and the alternatives.

Digitisation

Additional Comment: In our view, the setting up a digitisation centre plugs a significant gap in infrastructure.

On Demand Publishing

Additional Comment: The level of publisher involvement in the ODP area might be seen as a little disappointing although not unexpected. Certainly projects are making sterling efforts to address or work around publisher-imposed copyright restrictions. Although electronic publishing certainly encourages a hybrid academic/commercial publishing industry, it is unlikely that new small scale HE based initiatives will achieve the critical mass and integration required in the longer term. Consequently, as the commercial publishers are likely to be the lead players in the longer term it would have been advantageous to involve them more centrally in these pump priming activities.

More generally, it seems to us that what is being offered by the ODP projects is not real services but rather experimental services/scoping studies. If accepted as this, that is wholly appropriate, but, if not, there is bound to be disappointment. ODP continues to need more work on integration with teaching strategies and library reorganisation, and with publisher supply strategies.

Electronic Short Loan Collections

Electronic Short Loan Collections (continued)

Additional Comment: How electronic short loan collections differ functionally and technically from ODP is not entirely clear to us. We wonder whether the model here is too driven by existing patterns of provision. There is a need to look at not just what is being replaced, not just on what is being done, but on the emerging model of provision.

More generally, we can ask whether some of the ODP/Electronic Short Loan initiatives and EDD really make any sense outside Intranets when what the future may hold is national holdings/services? This leads back to the question of the individual IT strategies of HEIs - if what is required is short term low tech solutions surely they must be real institutional initiatives now. We have a persistent worry that projects are engaged in experimental services/scoping studies for an implicit set of future scenarios which are already outdated.

Electronic Document Delivery

Electronic Document Delivery (continued)

Additional Comment: We remain sceptical as to the economic case for the various EDD projects. Projectsí own forecasts of costs and charges, significantly lower than that of the British Library, are likely to prove somewhat optimistic. (The strategic case for encouraging competing solutions, remains, as does, more importantly, the catalytic effects on changing library work practices and organisational systems.) It is important that projects adequately model the cost, demand and price variables so as to produce realistic 'business cases' for continuing services beyond project lifetimes. EDD needs much more strategic action - if there is to be a credible integration project resulting from the current 'basket' of projects. Strategically, it seems to us that a lot depends on how JISC decides how it wishes to relate to the British Library. In reality there are probably only a limited number of strategic options, set up joint ventures with BL or set up one or more competing national services (say, one based on networked provision, the other based on a centralised service).

Images

Supporting Studies

Training and Awareness

Training and Awareness (continued)

Additional Comment: Now that they have been buttressed by appropriate arrangements for formative evaluation, the T&A projects clearly can make a contribution within their own stated project objectives. Netskills and Edulib provide useful support for general awareness raising and support for skills acquisition. However, T&A does suffer from insufficient targeting of user populations and insufficient coherence with project activities. Furthermore, T&A does not address training and awareness needs in the comprehensive manner intended by Follett (see Section 3).

We therefore repeat our comments made elsewhere that:

(a) it may be better in future to arrange such support work by commissioning or call to tender for a specific brief rather than by calling for proposals, and

(b) appropriate targeting of groups to attend the courses, and continuous formative evaluation during course delivery will be important if value for money is to be achieved from the current projects.

Although we recognise the many concertation events which have been held, if there is a coherent set of concertation and dissemination arrangements in hand then this is not apparent to us. It continues to be desirable that FIGIT establish a clear programme of communication within and beyond the programme through meetings, workshops and events. However, the current reorganisation of JISC which will include a Human, Cultural and Organisational Issues Committee (in which the eLib Training and Awareness projects will be located) should provide the strategic support to effect more coherent and cross linked efforts at changing organisational patterns and cultures with respect to ICT.

Related Initiatives:

We note here a number of related initiatives not included in eLib but addressed elsewhere:

Data Services

Included here is the ongoing extension of BIDS and other data services. Although they are central to the establishment of a national electronic library resource infrastructure, these services (we understand) tend to be based on non standard access methods and outdated proprietary systems.

Archives

Image based electronic archives will be an important eventual part of the electronic library. Current work continues to necessarily focus on cataloguing and preservation issues. There will be a need to relate emerging archival catalogue norms to the resource discovery approaches under development and promoted for other categories of electronic information.

Electronic Books

Although this has been mentioned as a possibly undeveloped area, given its relationship to within HEI purchasing, and teaching and learning strategies (courseware, course packs), it is not evident to us that electronic books requires R&D action or a HE wide strategy at this stage (although we are aware of the existence of views to the contrary). We note that a large scale, publisher led, European wide initiative in the area of multi-media publishing is ongoing (Info 2000), which may limit the additionality eLib could offer.

Bibliographic

We note that the CURL bibliographic data base is being developed as a national OPAC service.

6. Programme Future

Integration

As noted in the opening section, we only comment here on possible actions which could be considered to be within the remit of JISC/CEI, although these by no means exhaust the possibilities for action on developing the electronic library as a whole.

The electronic library is an attractive concept but is only likely to be realised as an institutionally pervasive concept. That is, library strategies can not be divorced from the internal organisation, research, and teaching and learning strategies of individual HEIs. (This could be said to be Follett's central insight as regards IT (see Section 3, above)).

If integration implies institutional change strategies, what then can the role of JISC be as opposed to individual HEIs? To provide attractive services which stimulate end user uptake and investment? To continue to support R&D with sometimes ill defined take-up paths? Or to pump prime HEI innovation by covering up front design and planning costs bearing in mind that whoever innovates first will bear a higher cost? Yet, promoting innovation is not simply a question of pump priming new services, R&D or end user investment, it is about creating new institutional/market configurations.

On the demand or user side this suggests institutional responses which seek to integrate the electronic library into institutional teaching and learning strategies, to provide research support services through the electronic library, and address the associated infrastructure, reorganisation and training implications. For example, HEIs may simultaneously invest in equipment, human resource development, reorganisation (process re-engineering), end user training, and refocusing of purchasing (to on-line sources and courseware), and in organisational intranets.

Disciplinary responses are also implied - new forms of communication, exploiting multi-media, combining multiple services, at a UK wide disciplinary level. For this there is a need to scale up and integrate diverse eLib projects on a disciplinary basis. Disciplinary strategies should include teaching and learning resources, on-line access, electronic journals, and various forms of communication (from pre-prints to electronic seminars to peer review). This requires resource commitments in learning resources, equipment, awareness raising, and services which offer/grant users a clear 'competitive advantage' (e.g. as chemistry data sets, on-line law reports, on-line archives, or dynamic multi-media resources do).

On the supply side, that is, JISCís side, what is needed is to continue to develop critical mass in supply - creating new service markets, achiving critical mass in purchasing, establishing a national document supply strategy, a national resource discovery strategy, a national archiving strategy and so on. This also requires the provoking of a market response from publishers and other content providers by creating and clarifying demand.

Thus, when we talk of 'integration projects' there is a need to address, to some extent simultaneously, individual institutions, disciplines and central JISC services as overlapping sets of services, disciplines and institutions. In short, this is the search for critical mass. This may take the form of funding a small number of integration projects (combining both institutions and disciplines in a way which involves all key aspects of HEIs - administration, personnel, research, teaching, library, computing, and management - and involving all key aspects of disciplines - research, teaching, publishing, communication, and all relevant departments, journals and societies). In other words, what is required now is not so much 'integration projects' which bring together various services (by and large the Web does this), but institutional projects extending from the electronic provision of internal information and resources and access to ëcentralí or remote services. This could imply scaling up existing initiatives horizontally i.e. across departments and faculties.

Cultural Change

The programme overall has a strong commitment to promoting 'cultural change' in the HE sector. It is probably fair to say that while FIGIT/JISC is highly sensible of the importance of cultural change to the diffusion and uptake of innovative technologies, it has been less clear as to what cultural change might entail in practice and what are the best mechanisms to promote it. For the purpose of this future orientated analysis, we propose to use one particular definition of cultural change, which we feel is most relevant to this context. Cultural change involves new frames of reference, new ways of acting. Cultural change results from actors acquiring new symbolic resources (cognitive frames/paradigms: concepts, knowledge, skills) in changed structural contexts (organisational contexts, work processes) where these symbolic resources are meaningful, deployable and operational. Cultural change is as much a result of structural change as a promoter - effective cultural change rarely happens without simultaneous change in structures.

Insofar as an expression of cultural change exists in the programme, it appears to be based on changing individualsí behaviour and orientations through providing information, exposure and/or training. However, an adequate conception of cultural change needs to recognise that cultural change must be based as much on changing the contexts (work systems, work practices, job definitions) in which individuals operate as on retraining or reorienting the individual. In fact, cultural change requires a dialectic between developing individuals so that they can fulfil new roles and changing contexts so that individuals can utilise and express new understandings and skills.

The promotion of cultural change in eLib has been seen as driven by two main motors, firstly, mobilisation of actors in the community to change the structure of their activities. This has been based on the strategy of funding a large number of multi-institutional partnerships. Secondly, training and awareness activities (providing new symbolic resources). This has included a combination of targeted training initiatives and awareness raising and consultative/concertation events and various dissemination activities, such as Ariadne, and most importantly, the eLib Web site and lis-elib. We have suggested above (under the heading ëTraining and Awarenessí) the limits of these two mechanisms. Here we wish to consider more generally the logical scope for promoting cultural change given by the programme structure, the extent to which this scope is exploited by programme activities, and the extent to which the programme structure in itself limits the scope for promoting cultural change. We do this by considering five aspects of cultural change which implicate all eLib projects, namely, training and awareness raising; communication and dissemination; organisational commitment; scale/critical mass; and integration/convergence.

Training and awareness raising: In principle, it can be argued that every project (or set of projects) should have its own reinforcing training and awareness activities. This indeed is the case with a number of the projects with operational services (e. g. SOSIG), but is by no means universal. As many projects are still in the development stage this is understandable, what is of concern is that coherent training and awareness strategies have not been planned as part of plans for product or service roll out.

Communication and Dissemination: Linked to the issue of T&A is that of communicating project results and transferring project technologies beyond the project. The programme has relied too much on the 'beacon' model of imitation, and too little on active strategies to transfer technology through widening the network of utilising and participating institutions, e.g. marketing/offering complete technology system plus training, consultancy and support packages.

Organisational commitment: Achieving cultural change requires, simultaneous to the introduction of the new technology, an organisational restructuring of operations, and a reskilling of staff so that they can fill their new roles in the new structure. Developing the new technology is only one part of the change process, and will not of itself bring about cultural change. (Current projects are only beginning to appreciate that there must be a managerial commitment to reorganise work processes/ways of doing things.)

Scale/critical mass: HEIs or disciplines can not be expected to initiate complex change processes until they can be assured of tangible benefits. In practice, due to the complex set of simultaneous changes which must be made in staffing, training, work processes and equipment, this often means shifts in whole tasks/modes of delivery rather than incremental change. Cultural change effects can only be gained by significant restructuring, not by small scale pilot demonstrations (which promote learning and raise awareness but do not achieve cultural change). We can reflect with hindsight that projects at proposal stage might have been encouraged to enlarge rather than to curtail their spending.

Integration/convergence of projects: Major innovation takes place at the level of sectors rather that of individual organisations. If individual project innovations are to prosper, be taken up and used in the context of new practices, they must mesh with and reinforce other innovations in the sector. Projects must be cognisant of where they fit in wider change processes, that is, what processes of change do they allow to happen and what else needs to happen to allow this contribution to succeed. eLib has a solid basis for such integration/convergence, what is now needed is a (funding) mechanism to encourage such behaviour.

Other Specific Future Actions

Teaching and Learning

A key insight in discussing information provision in HE in recent years has been the idea of 'resources for courses'. The idea of providing tailored, targeted resources for courses has been long established in open and distance learning, in the idea of ' course packs' and 'readers', and in the ubiquitous library short loan and photocopy collections. Recent electronic expression has focused on incorporating multi-media resources as part of ordinary library provision, especially as regards providing computer based learning materials which address core or common curriculum elements across a wide range of courses and institutions.(24)

In eLib this has been addressed in the linked areas of ODP and electronic short loans, where as we have seen, ODP in the first instance, in many projects, has the form of electronic course packs or readers. However, electronic initiatives in this regard have been limited in two regards. Firstly, the use and deployment of courseware and electronic sources is often not integrated properly within a broader pedagogic (teaching and learning) strategy, and, secondly, the provision of electronic texts (images, etc.) is not sufficiently comprehensive enough to be the main or sole source for course materials. These deficiencies are not likely (nor, mainly, even planned) to be overcome in the current set of eLib pilots, although the technologies produced will (it is hoped) in due course be scaleable at an institutional level to level of provision which can support pedagogic strategies for courses based on a core of electronic resources. What is now needed are some pilot experiments in offering sets of courses supported with a critical mass of electronic resources (e.g. including all reading materials, lecture notes, assignments, exam questions) or even without the provision of paper alternatives. This obviously requires a high level of institutional commitment to change, central involvement of lecturers, and assurance of adequate, available equipment (PCs, printers) for students.

Intranets

The rise of the Intranet browser reinvigorates the concept of the electronic office or more precisely the electronic filing system. Many of us are now familiar with the idea of dynamic updating of information sources through our use of the Web (particularly as regards help systems and manuals, and catalogues). In large organisations, with a continuous need for amending and updating large document bases, the browser based intranet is seen as the bridge between databases and word processed documents. HEIs, as organisations, with ever changing information (on courses, on staff, on timetables, on procedures and rules, on equipment and infrastructure) and a dispersed organisational structure, are particularly suited to the adoption of intranets.

However, there are limits to the extent to which it makes economic sense to incur the costs of electronic systems while continuing to maintain paper based systems of storage and access. Paper will continue to be used indefinitely as a preferred medium for reading/working with information and for personal storage and transport. But at an organisational level, if efficiency and economic gains are to be made, a transition may be necessary from one basic system of information handling to another. This would require root and branch change: new work processes and procedures, new equipment and infrastructure, and training and reskilling for all. Such a commitment to institutional change would have to be led from the top and involve every element of the innovatory institution.

Developing such a socio-technical system requires extensive design and planning work, particularly for the first innovating institutions. It is work that institutions might contract out to a commercial systems house, but it may be also something JISC may wish to fund on an experimental or pilot basis in a small number of enthusiastic institutions. Once such an investment and commitment was made by an innovative HEI the opportunities to use the same systems (if successful) to support teaching and learning resource provision should be obvious.

Institutional Change Projects

As with the establishment of intranets, integrated systems for electronic supply of resources for research and teaching, require, if they are to be scaled up in practice, major institutional commitment. If the necessary learning and changes in practice made by academics, lecturers, students and librarians, and the associated investments in infrastructure and support, are to be worthwhile, critical mass must be achieved. But changes on such a scale involve a steep learning curve and considerable potential disruption and short term, start up resource commitment.

Learning how to introduce such change can only be based on actual pilot projects involving whole institutions. The challenge here is not simply to design the technologies but to orchestrate the organisational change process. There will be considerable costs to those institutions who are the first to innovate, and a great deal of learning accruing to the whole sector, which could be seen as justifying central funding of the planning, design and evaluation components.

(Of course, the same logic can be applied to disciplinary change projects).

Multi-Media Use and Disciplinary Environments

The multi-media content of the EJ area is somewhat light. More work needs to be done on the ordering of whole electronic environments (i.e. which integrate resources delivered through ANR, EDD, EJs and so on), at a disciplinary or multi disciplinary level. A way forward here may lie with funding a small number of large scale discipline based initiatives.

Information Resource Discovery and Management

Associated with the need for broader disciplinary initiatives, more fundamental research on resource discovery, organisation and usage along the lines of the US digital library projects may need to be undertaken. This requires large test beds of materials and extensive piloting and iterative development. This could be based around pre-existing large data sets and gateways (e.g. in chemistry or economics). (Whether this is a proper task for JISC may be debatable. Perhaps it is the responsibility of the research councils, although JISC would provide the 'test bed'.)

Training and Awareness

Training and awareness needs to be much more closely associated with actual projects and initiatives. It should be commissioned by tender as targeted support actions with well defined objectives and target groups. Examples might be, support for designing and establishing ODP for teaching purposes, intranets, EJs etc. Much T&A course content could be based on actual previous eLib learning and experience, i.e. T&A products should be an output as well as an input to projects. One output of CEI could be a stream of training products for CALT.

JISC promotes cultural change by providing facilities, services and opportunities. In keeping with this, every T&A action should be linked to a specific service or activity, either general services to support specific types of HEI actions, or directly linked to usage of a JISC service (e.g. in eLib T&A actions should be directly linked to area activities).

Evaluation

eLib is attempting to promote innovation, mobilisation and catalytic effects through a number of mechanisms. It will be necessary to evaluate how successful they have been (it is not enough to 'do something' on evaluation, on PM, we have to see how has it worked in practice) and what can be learnt for future JISC development initiatives. Among the things which will need to be reviewed in the concluding summative evaluation will be:

• how appropriate was the scale of projects and what problems arose from sheer numbers and in scaling up?
• how successful have the project selection, management, steering, and evaluation mechanisms been?
• how adequate were the communication, T&A, and dissemination mechanisms?

Promotion of Good Development Practice

Much of the most valuable content of the evaluation workshops arose from addressing pressing gaps in project organisation which were not strictly about evaluation issues, but where the requirement for evaluation directed systematic attention which was hitherto lacking. Most notably, these included issues of; user centred and structured design and development methodologies (addressed through the theme of evaluating interfaces and pilot tests); project management (addressed through the theme of evaluating project organisation and utilisation of evaluation results), and business and financial planning (addressed through the theme of using evaluation to inform planning and exit strategies)(25).

It seems clear to us, even at this early stage, that in subsequent development programmes JISC needs to ensure that:

a) it requires much clearer documentation, in project proposals, or in any case as a condition of funding, on the design and development methods to be employed, the project management techniques to be used, and the business techniques deployed to develop exit strategies, and that projects have staff or consultants available to them who are competent to fulfil these roles, and

b) that adequate support (guidance and training), is provided to projects who continue to need it along similar lines to that provided for project management and evaluation in the current programme.

Having said this, we also recognise the need for programmes such as this to be inclusive rather than restricted to a small number of composite experts. The eLib programme is currently a site of huge learning gains for the programme participants. As an initiative that now spans the HE sector and involves directly several hundred individuals, this intra programme learning may be as important as any current leverage upon HE institutions and their libraries. In this sense, the whole programme is an awareness and training activity whose significance may become apparent in time as project individuals consolidate their influence to extend and institutionalise ICT within HE.

(1) Higher Education Funding Councils’ (HEFC) Joint Information Systems Committee.

(2) Follett Implementation Group Information Technology, a JISC sub-committee.

(3) Higher Education Institution

(4) Higher Education Funding Councils’ (HEFC) Joint Information Systems Committee (JISC) Committee on Electronic Information.

(5) Higher Education Funding Councils’ (HEFC) Joint Information Systems Committee.

(6) Follett Implementation Group Information Technology.

(7) Higher Education Institutions

(8) See, Tavistock Institute documents, Evaluation of the Electronic Libraries Programme: Phase One, A proposal submitted to the Programme Director, (August 1995) and Evaluation of the Electronic Libraries Programme: Evaluation Framework for eLib, (January 1996).

(9) eLib Programme Director.

(10) JISC secretariat.

(11) JISC Secretary.

(12) Particularly, the recent JISC papers, Exploiting Information Systems in Higher Education: An Issues Paper (March 1995) and JISC Five Year Strategy, 1996-2001, (May 1996).

(13) In the areas of Digitisation, Pre-Prints, Quality Assurance, Electronic Short Loan and Images.

(14) HEFCs’ Teaching and Learning Technology Programme (TLTP).

(15) Non-Formula Funding of Specialised Research Collections in the Humanities: Archives Sub-Programme.

(16) We are grateful for the comments and criticisms provided by Lynne Brindley, Andrew Jordan, Nicky Gardner and Chris Rusbridge. Responsibility for remaining weaknesses, errors and opinions remains with the authors.

(17) The full list of areas were initially: Electronic Journals, On Demand Publishing, Electronic Document Delivery, Digitisation, Awareness and Training, Access to Network Resources, Supporting Studies. These were followed by: Images, Pre-Prints, Quality Assurance and Electronic Short Loan.

(18) One of these projects was subsequently terminated.

(19) The decision-making process surrounding the commissioning of the imaging and digitisation areas where a number of approaches were tried is a good case in point.

(20) For example, SEREN was formed as a composite project from two (plus one) discrete proposals.

(21) A national digitisation centre will shortly be funded.

(22) In practice these descriptions will be almost immediately revised and amended in the light of the imminent receipt and analysis of the projects’ annual reports.

(23) Particularly through the evaluation workshops and clinics.

(24) One emerging consensus from the TLTP Programme is the particular suitability of courseware for first year ‘foundation’ courses which tend to incorporate similar elements in most undergraduate curricula.

(25) On occasion, the workshop staff had the impression that when they referred to standard good practice in information systems design and development, they were introducing wholly novel concepts to some of the participants in the workshop.

[Index of Tavistock Reports and Papers] [Meta-index of Papers, Reports and Circulars]

Created using SoftQuad H.i.P. Version 1.0.2.17
Web page maintained by Isobel Stark of UKOLN
Last updated on 04-Apr-1997

The Electronic Libraries Programme (eLib) was funded by the Joint Information Systems Committee (JISC)
Page version: 1;
Web page maintained by UKOLN Systems Team and hosted by UKOLN - feedback to systems@ukoln.ac.uk .
Site last revised on: Tuesday, 24-Nov-1998 14:21:06 UTC
DC Metadata