Lessons from Phase 3 of the Electronic Libraries Programme
“For the next ten years, if it works well, is reliable, and you know how to use it, it is obsolete.” John V. Lombardi, 2000.
This paper attempts to summarise some of the main ‘lessons’ which have emerged from Phase 3 of the JISC-funded Electronic Libraries Programme (eLib). It is primarily a view from the inside of eLib. It incorporates contributions from eLib3 project managers and other stakeholders. It is not a formal evaluation of the programme; this will be done in Spring 2001. Rather it is an attempt to synthesise some of the key issues which are important to those involved in the ‘hybrid library’, ‘clumps’, and ‘projects to services’ strands of eLib3. Some comments are also made about ‘digital preservation’ (the remaining strand of eLib3). This does not mean that eLib3 stakeholders will all necessarily endorse every point, but that at least some of them have identified many of the points outlined as important and worth communicating. The lessons below are followed by recommendations for further work.
1.1. An iterative development cycle is often the best way to make rapid progress.
This involves an ongoing cycle of design – build – evaluate – redesign. As well as leading to more rapid development, an iterative approach can be a good way of engaging users and others not directly involved in the project. It can also help to ensure that emerging products are more in-line with user needs. There are many successful examples of this in eLib3.
1.2. Demonstration is better than explanation.
It is often important for a project to have something to show early in order to communicate the issues to stakeholders.
1.3. Exit and scaling strategies need to be built into project plans.
These need to be fully developed as the project progresses so that as the project funding ends, the project outcomes can still be taken forward. Some practical support and advice in planning for project exit should be provided by the programme office. Where project outcomes are likely to have wide significance, JISC should provide more support to help projects transform into services (see below).
2.1. Project management training is essential for new project managers.
The training given to eLib3 managers was well received and found to be very useful. This should be repeated for other new JISC-funded project managers and should be arranged so that they can all attend.
2.2. Elaborate formal project methodologies can be difficult to implement successfully.
Whilst it is useful to be informed by formal project methodologies, following a system such as PRINCE2 was thought to be ‘over the top’. It is however important to have an informal methodology and to follow sensible project management protocols. A systematic structure for documenting the project (producing specifications, reports etc.) is important, and can be usefully informed by formal project methodologies.
2.3. Detailed project planning is nevertheless essential.
eLib3 projects were required to produce formal project plans within three months of project start-up. Although this creates a heavy workload, it is ultimately very useful. A well-developed project plan is useful throughout the life of a project, especially if it is kept up to date.
2.4. The roles and responsibilities of the project manager, management group and steering committee should be clearly defined and publicly agreed as part of project start-up.
In particular, the question of who is ultimately responsible for the project should be made clear. The decision-making and decision-ratifying processes should also be formally agreed by all of the parties involved. In addition, responsibilities for actually implementing decisions on the ground should also be clearly understood (especially in a multi-site project).
2.5. eLib project managers have considerable experience of running projects which should be passed on to new JISC programme managers.
It was thought that a meeting could be organised between eLib and JISC 5/99 managers to surface these lessons. The possibility of ‘mentoring’ arrangements might also be considered in which new project managers have access on an informal basis to those who have worked in completed projects.
3.1. Recruiting and retaining project staff (especially technical staff) is difficult, particularly with short-term contracts.
Recruiting and retaining skilled staff is critical for project success but can cause major problems, particularly in the last year of a project. Secondments of staff may be useful but there is still the problem of finding these people roles beyond project closure. Many staff do not want to return to their old job after the experience of the project.
3.2. There are both pros and cons in assigning existing staff (especially technical staff) to projects on a part-time basis.
The most of obvious pro is that existing staff are already familiar with the existing systems and services. The most obvious con is that they often have more pressing demands on their time than development work. If existing staff are to be used part time, their time commitment to the project should be clearly agreed and monitored.
3.3. Project staff can act as a valuable development team for the library as a whole – keeping up to date on new thinking and developments and informing colleagues.
Project staff can often contribute to the host organisation by promoting innovative atmosphere. It is felt that libraries should try to find ways to maintain a group of staff who have a development brief in their organisation.
4.1. Project managers often need training in financial management.
This can often be true in general terms but also in terms of institution-specific financial procedures.
4.2. Programme Office requirements for annual reports can sometimes be out of step with institutional reporting timetables.
Many project managers attempt to overcome this by keeping their own independent financial records rather than relying on institutional information. This can be time-consuming.
5.1. The balance between ‘off-the-shelf’ and ‘custom-built’ software tools needs to be considered carefully.
‘Off the shelf’ software solutions are often easier to obtain but may not always fit the requirements exactly. ‘Custom-built’ tools fit the requirements but are time-consuming to develop and require the commitment of ongoing support.
5.2. Project development paths can often get out of synchrony with wider organisation or institutional paths.
Ultimately, every project has to produce deliverables. Sometimes, in order to get the job done, projects may have to depart from long-term strategies being adopted in their host library organisation or institution. It is also possible for project and organisational/institutional strategies to drift apart over time. A project strategy which had institutional support at the beginning of the project may not have it by the end of the project. This can cause problems for the long-term integration of the products delivered by the project within the institution.
5.3. The relationship between hybrid library developments and the local OPAC (and Library Management System in general) remain unclear and need to be further considered.
It is unclear whether the local OPAC should be ‘just another database’ provided by the library or whether it should be the core the hybrid library service. Some projects have suggested that the hybrid library system should be a wrapping for the OPAC (amongst other databases), whereas others have suggested that the OPAC itself should be gateway to all services offered by the library service.
5.4. The basic approach of hybrid library development of searching a number of different databases through a single user interface has been questioned by some.
A significant minority of eLib project staff have observed that there has been an increasing movement in the information world towards branded services on the Internet and that this is a difficult trend to resist. It has been suggested that users like to know exactly what they are searching rather than being in a position where they send a query to an anonymous amalgam of services behind a single interface. To others, the benefits of cross searching are self evident. Cross searching is seen by many as valuable in particular for the first stages of searching. More detailed searches may require the user to drill down to specific services. It is of course essential that any results from cross searching should be reliable. It should also be obvious to users where results are coming from.
5.5. JISC-provided e-services should have a more unified front end.
The increased co-ordination represented by the development of the DNER Office is an opportunity to attempt to create a more coherent front end. At the very least, there should be a single front door to available resources. Ideally, the imposition of coherence should go further by encouraging data providers to allow richer access to their data through protocols such as Z39.50, LDAP etc.
5.6. DNER services should be delivered in such a way as to enable institutions to incorporate services within their own local environment.
The RDN-i is a promising example of how DNER data (from the RDN) can be delivered within a local interface. Developments of this kind should be encouraged so that the DNER becomes a service enabler as well as service provider.
5.7. The hybrid library should be integrated into the emerging MLE, but the way the two fit together is not yet clear.
When eLib3 projects began hybrid library systems were seen effectively as standalone library-based developments. It is now clear however that the hybrid library needs to mesh with emerging Managed Learning Environments and student portals. JISC should support further development work in this area.
5.8. Authentication and authorisation are key elements to successful hybrid library development but remain major problems. The development of a national solution which can be customised to local needs is seen as the best way forward.
ATHENS was seen as too inflexible to have been used by any of the projects. It is hoped that SPARTA will be easier for institutions to customise to their own local needs. In the meantime, a number of projects have developed a successful ad hoc local solutions to authentication but it is unclear how transferable these are.
5.9. Security and user friendliness are often in inverse relationship to each other – the two need to be carefully balanced.
It must be realised by system designers that the more secure something is the less user-friendly it normally becomes. Users are often put off by authentication challenges, especially if different services require different user responses.
5.10. Off-campus access to hybrid library facilities is crucial.
Users of all sorts are increasingly expecting off-campus access to resources. It is important for libraries to consider both the technical and licensing issues to allow this.
5.11. Key elements of personalisation conceptualised and developed in eLib projects are becoming important in wider institutional MLE/portal development.
MLE developers can learn from the hybrid library projects, especially in the areas of interface design and personalisation. Other JISC projects in the area of MLE development can learn from eLib outcomes.
5.12. Z39.50 has been shown to be implementable by a number of the clumps and hybrid library projects.
Z39.50 has been made to work within the project environment (although this has required more effort than expected). Z39.50 has been used not only for searching library catalogues but other datasets as well. However, problems remain in important areas: holdings data, and serials information to name but two.
5.13. The long-term future of Z39.50 for creating reliable virtual union catalogues or cross-searching facilities is still not certain.
The implementation of Z remains a very complex task. This is especially true in a context where the underlying data being searched (often MARC records) have been constructed inconsistently across different databases. Whilst pragmatic solutions to many such problems have been worked on a project basis, the long-term wider implementation of Z remains in doubt. More work, especially in Z mapping techniques, is required. Truly dynamic clumping in particular remains problematical.
5.14. Projects can play a valuable role in shaping developing technical standards.
This has been true in areas such as Z39.50 for Music, the Bath Profile and collection level description.
5.15. Digitisation is complex and resource-hungry.
Scanning is itself complex. But scanning is only part of a larger workflow involved in digitisation. If digitisation is to be carried out on any sort of scale it needs to be carefully managed and properly funded.
5.16. Preservation should be built into digitisation planning and processes.
This adds to complexity and costs but is nevertheless essential.
5.17. “Our capacity to collect and rely on digital materials and systems still outweighs our understanding of what it means to preserve them into the future.”
This fact (expressed here by Kelly Russell) means that particular notice should be taken of the technical and management issues associated with digital preservation when creating or acquiring e-materials.
5.18. Electronic short loan services are still too expensive to be scaled-up in institutions.
Publishers’ charges are still unrealistically high for electronic short loan services to be anything more than pilots. This needs to be clearly communicated to the key players.
6.1. Evaluation should be an ongoing process.
The division of evaluation into formative and summative sections is a useful one to encourage ongoing (as well as retrospective) evaluation activity.
6.2. A ‘neutral’ third party evaluator can often provide useful expertise and insights into project development.
Many projects have worked well with external evaluators. The outcomes of external evaluations are often more informative for the project and outsiders.
7.1. Dissemination activity should take place throughout the life-time of a project.
Dissemination may take a number of forms and be aimed at different audiences but it is essential that it is carried out from the outset. Projects can often establish a ‘brand recognition’ which can be useful for the later stages of projects when ‘hard results’ are disseminated.
7.2. Joint dissemination activities are often successful.
eLib3 projects carried out a significant amount of co-ordinated dissemination work (including articles, conferences etc.). The hybrid library projects also worked out a series of shared presentation slides about all of the projects which could be used by presenters at different events as an agreed way of talking about the projects.
8.1. It is important for project partners to agree on internal IPR and copyright issues early.
Many projects have generated learning outcomes and products which need to be disseminated. Some have involved innovations which are potentially marketable commercially. The IPR issues need to be agreed by all partners.
8.2. The interests of the HE community can best be represented on a national level by JISC and Universities UK to the CLA and legislators.
A number of projects have worked with the CLA and have contributed to the debate on IPR and copyright. Whilst these individual projects may have some impact, activities of national bodies are most likely to make a difference.
8.3. Pressure needs to be applied to e-publishers and data providers to allow off-campus access to their products as the norm.
Many licences are unnecessarily restrictive and out of line with modern learning and teaching developments. This should be made clear by JISC-supported negotiating agencies.
8.4. Licenses should also include provisions for long-term access to and archiving of materials where appropriate.
Many licenses do not explicitly address the issue of long-term access or preservation. Where possible JISC negotiating agencies should take this on board.
9.1. Distributed projects involve a considerable management overhead; this should be built into project proposals and plans.
The vast majority of eLib3 projects involved consortia of different HEIs. Keeping these consortia intact as well as driving the project forward is a considerable undertaking. Partners need to be clear about their ‘shares’ of both the benefits and the responsibilities of work input.
9.2. Cross-institutional project teams in particular require careful management.
Across different institutions staff may have different terms and conditions, different priorities and different management pressures. It can sometimes be difficult to ensure they work in a co-ordinated way. The project management protocols should include specific reference to how these internal project management issues should be dealt with. Where possible it can be useful to involve the project manager from one institution in drafting the job description and participating in the recruitment of project staff for another.
9.3. Successful consortia are usually those which are composed of ‘natural partners’ rather those where partners have been brought together as a ‘marriage of convenience’.
A number of eLib3 projects were based on existing consortia. This meant that those involved tended to work together well and were more likely to have a clear view of how project outcomes could be taken forward after project closure. Other projects were composed of institutions with obvious natural interests. These situations work better than bringing institutions together simply to gain project funding.
9.4. All project partners should be encouraged to take ownership and responsibility for project progress in their various institutions.
In a multi-site project it is essential that each partner takes responsibility for taking its part of the project forward (whilst still, of course, maintaining contact with the project manager). Partner sites can sometimes ‘sit back’ and wait for things to happen at the lead site, rather than taking responsibility themselves for implementing their part of the project.
9.5. Where possible the relationships between partners should be documented in a memorandum of understanding.
This helps to ensure that the partners have clear views of their respective ‘shares’.
9.6. The success of partnerships with the public library sector has been mixed.
Some projects have found the interaction between the HE and public sector has resulted in a useful exchange of views. Others have found that differences in the funding, culture, and technology made real collaboration difficult.
9.7. Working with commercial partners requires significant time and effort.
These partnerships need to be very carefully managed. It can take time to establish relationships and shared ways of working.
9.8. Partnerships with commercial providers are potentially beneficial but need to be clearly defined early – service level agreements and contracts including penalty clauses are often necessary.
Some projects have found that commercial partners have been unable to meet their expectations. Deliverables have been late or incomplete or both. HE and commercial providers often have very different working practices and strategic priorities. Clear partnership arrangements, contracts and service level agreements (including penalty clauses for non-delivery) have been suggested as possible ways to overcome some of the problems. Most of all, good communication between the partners is essential to ensure that their expectations of project development are shared ones.
9.9. Commercial providers are often more willing to work with an HE project when it has a ‘secure’ future.
The opposite is also true. Commercial companies are often reluctant to ‘buy into’ projects which are short-term activities and do not have a clear role in future JISC strategy.
9.10. A number of the projects have successfully negotiated with data providers to gain access to their data for cross searching. It makes sense for this in the long term to be carried out on a national (rather than project-by-project) basis.
National agencies such as the DNER can ensure that the approach to data providers is consistent. Data providers are more inclined to work with one large organisation than with several small ones.
10.1. Cultural issues are as important as technical ones.
Both within the library and amongst its users cultural issues are crucial and need to be addressed. People often have habitual ways of working (and thinking) which are not easily shifted.
10.2. Technical developments need to move in step with policy developments.
For example, greater cross searching leads to greater demand for access to resources. The policies in place in libraries need to be developed to take account of this.
10.3. Non-project staff within the host organisation need to be kept informed about project progress and encouraged to participate in all possible ways.
Projects do not just need to disseminate widely on a national level but also communicate within their own organisation. It is important that project staff do not become isolated in their own organisation. Non-project staff can play a valuable role in contributing to and promoting project developments. They should be given formal opportunities to do so through seminars, prototype testing etc.
10.4. The support of senior LIS and institutional managers is essential.
As the HyLiFe project has pointed out, it is often useful for projects to have a ‘champion’ at senior levels of the institution. The support of a manager with ‘clout’ can help to ensure the project is allowed to develop rapidly and also to help ensure that project outcomes are embedded in the organisation.
10.5. Project overheads (secretarial support, accommodation costs etc.) are often significant.
These are often not costed in project proposals but can be significant for the host institution.
10.6. Considerable commitment is needed to move from project to service.
There are major staff training, user education, marketing and support implications when products are rolled from a project to become a service. These should not be underestimated.
10.7. A successful project can create a ‘feel good’ atmosphere in its host organisation.
Successful projects are good for the organisation as whole, especially if local staff own the project and its outcomes.
11.1. User expectations need to be managed.
Expectations can often be unrealistically high, especially after seeing a prototype. It is essential to communicate the limitations of prototypes and length of development timescales.
11.2. Difficulties can arise in getting users to be involved in the design, testing and evaluation of products.
Whilst all developments in eLib3 aimed to be driven by user needs, it was often difficult to recruit users to participate in project development and evaluation. In particular, it was difficult to find users who were prepared to participate over the life-time of a project.
11.3. Users are reluctant to learn new systems unless there is likely to be a considerable gain. There needs to be a ‘critical mass’ behind developments.
It is often difficult to encourage users to try out new developments. There is a natural (and rational) inertia. Developments are usually only successful if they make a real difference to users’ work.
12.1. There are problems connected with delivering services to an entire community when there are differing needs within the community. This is likely to become more significant with the inclusion of FE within the JISC arena.
It perhaps needs to be recognised that each service will not be of use to all users in the HE/FE community.
12.2. The development of the hybrid library requires LIS staff to work in partnership with academic colleagues.
LIS staff are increasingly becoming more directly involved in learning and teaching issues. Communication and liaison mechanisms should be strengthened. LIS staff should ensure they have an input into learning and teaching and research strategy.
12.3. Many innovative developments in the area of online ordering of materials and e-commerce can be held back by out-dated financial management and auditing procedures.
Existing financial procedures in institutions are often excessively paper-based.
12.4. Partnerships with other support services, especially the computing services, are crucial for the development of the hybrid library.
The hybrid library is in many respects ‘convergence in action’. The library and computing service need to work in partnership to deliver effective services.
13.1. JISC calls should be more focused and be linked to a clear strategy.
Some JISC calls have been vague. This often means that project outcomes are equally vague. It may be better for calls to be more specific and in line with a specific development strategy.
13.2. The bidding process for project funding should be made more transparent and institutions allowed to learn from the bidding process.
In particular, it should be clear against what criteria proposals will be measured. I would be most helpful to unsuccessful applicants if they could see referees’ comments (suitably anonymised).
13.3. The roles and responsibilities of JISC, the programme office, steering committees and projects in relation to each other need to be clear.
For example, projects need to be informed about what level of support they can expect from the programme office. Projects also need to be clear about what is formally required of them by JISC and the programme office in terms of documentation (whether or not they should submit copies of evaluation reports, minutes of meetings etc.) and what documentation they can expect in return.
13.4. The eLib Programme Office approach of achieving a balance between ‘control’ and ‘freedom’ in relation to projects was largely successful and should be continued in future programmes.
The eLib Programme Office maintained a good balance between, on the one hand, carefully monitoring project progress against planned milestones, and on the other hand, allowing projects the freedom to pursue new objectives in the light of changing circumstances. This worked well in eLib and would probably work in other similar development projects. It is a good compromise between the inflexible approach exemplified by say the EU and the more laissez faire approach of the research councils.
13.5. Projects appreciated the support of the eLib Programme Office and felt that it was closed down too early.
Programme office staff should be employed for as long as the projects themselves.
13.6. The importance of face-to-face contact should not be underestimated in the context of programme management.
It was thought to be helpful if programme staff could try to visit projects on regular basis. It is often possible to learn more this way than from reading reports or listening to presentations. Visits may uncover hitherto unthought of points for celebration and/or concern.
13.7. Many projects felt they had inadequate resources to develop services within JISC timescales.
This was especially true where negotiations were required with external stakeholders. Projects are often given less money than they had bid for but still expected to achieve the same outcomes.
13.8. The programme-based milestones of annual reports can often be out of synchrony with project cycles.
This can often cause considerable administrative work for projects. The current strategy implemented for JISC 5/99 projects (where reports are shorter but more regular) is a better way forward.
13.9. JISC-funded advisory services should be available to take a more active involvement in projects, particularly in technical areas.
There has been little formal interaction between eLib projects and JISC advisory services (such as Web Focus, Interoperability Focus and TASI). A two-way dialogue between projects and advisory services should be encouraged. Each can learn lessons from the other.
13.10. There should be a synthesis of technical developments within projects produced by a supra-project unit (perhaps an advisory service or the programme office).
This was done informally by eLib3 projects with developments such as the joint clumps page and the hybrid library projects search engine but these were not as effective as they might have been. Ideally, the synthesis work should be carried out by a third-party.
13.11. It is essential for the programme as a whole (as well as individual projects) to have dissemination and evaluation policies.
The programme office can do valuable supra-project dissemination. Mechanisms should be found to ensure that project and programme dissemination activities work in concert.
13.12. The programme office should bring similar projects together early in their life to work out any synergies that may exist.
The earlier this is done the better. Enforcing collaboration between well-established projects can be problematical.
13.13. Informal inter-project collaboration worked well within eLib3 and should be encouraged in future programmes.
eLib3 project managers met regularly to discuss developments of mutual interest. This was very useful and should be encouraged.
13.14. There was some cross-strand collaboration but this was more limited and could have been improved.
Collaboration between different strands of eLib3 could have been improved.
13.15. Senior JISC players should maintain support for the projects throughout the life of the programme.
The active and vocal support of senior managers in JISC is important for the projects. There is a feeling amongst eLib3 projects that JISC managers were ‘moving on’ long before eLib projects were due to be completed. Talk of eLib being “dead” 18 months before many projects were due to end was felt by many project staff to have undermined their credibility.
13.16. JISC should provide more support to assist in the move from project to service.
This should include funding but also advice, contacts etc. The US model of ‘incubators’(where services have free accommodation, equipment etc for a limited period before being expected to operate independently) may be useful.
1.1. It is hoped that the lessons emerging from eLib3 outlined here will be taken on board by JISC and members of the HE community.
In particular, it is recommended that JISC should consider how issues raised in this report can be taken forward and incorporated into future planning and activity.
1.2. Ways of passing on the practical lessons from eLib to new JISC development projects should be considered.
These might include a structured meeting of project managers, and/or informal mentoring arrangements.
1.3. General consideration should be given to what ‘hybrid-library-like’ services can be incorporated into the DNER.
This applies to ‘hard services’ but there are also general issues associated with hybrid library thinking which should be considered.
1.4. eLib3 projects should be formally consulted in the DNER technical review.
The DNER technical review is an excellent opportunity to build the lessons of eLib3 projects into JISC strategy. Project staff should be formally consulted as part of this process. Ideally, projects should be given an opportunity to input ideas early as well as comment on the draft report.
1.5. In future programmes, consideration should be given to ways of further clarifying the relationship between the programme office and projects.
For example, it should be clear to projects how they will be supported by the programme office in terms of practical help, co-ordinating events and providing guidance. The formal documentary deliverables (such as reports and minutes) required of projects by the programme office should also be defined.
1.6. The programme office should consider ways to synthesise technical developments emerging across different projects.
This work should be accessible to the HE community in general.
2. Future initiatives requiring JISC funding
2.1. Consideration should be given to the possibility of establishing a ‘Hybrid Library Focus’.
This would probably take the form of a post, like the Web Focus. The post would operate on the interface between the DNER and institutions, and might usefully look at working up a formal hybrid library specification. It might also be used to give further consideration to the relationship between the hybrid library and the institutional OPAC.
2.2. The DNER Office should further investigate a number of relevant technologies which may contribute to greater coherence at the national level.
These include OpenURLs, context-sensitive linking, and DOIs. Emerging commercial products which incorporate these technologies should be evaluated.
2.3. Further research and development are required on the relationship between the hybrid library and the MLE.
Ideally, JISC should fund further research in this area in addition to the small number of projects funded in JISC 5/99 which touch on this issue.
2.4. Further research and development work should be carried out on Z39.50.
This should look in particular at the viabilility of using Z39.50 in the DNER and for large-scale services. The various approaches of the clumps projects might usefully be compared and analysed. In addition more work needs to be done in assisting institutions to implement Z.
2.5. Work on mapping different implementations of MARC against Z39.50 requirements is required.
Research work on the extent to which Z could be implemented widely across the community, bearing in mind the wide variations in the implementation of MARC should be carried out. There also may be an opportunity to promote more standardisation in the way the MARC is used in the future.
2.6. JISC should promote the development of a collection level description standard.
There is a danger that CLD is already beginning to be implemented in a wide range of ways. There is an opportunity now to achieve greater standardisation.
2.7. The question of interface design, particularly in relation to cross searching needs investigating.
eLib3 projects have done some work in this area but a more systematic study on how users cope with cross searching and how screen design can help is needed.
2.8. JISC should give more consideration to mechanisms for helping projects transform into services.
These might range from formal incubator services to simple ‘how to’ guides.
Author: Stephen Pinfield
Version: 2.2
Status: For JCEI consideration
Date: 23 January, 2001