In this concluding section of the overview report, we offer some reflections on the annual reports, drawing attention to what we perceive to be the strengths and weaknesses in the development and implementation stages of projects during their first year.
Experience of the development process
It is evident from the annual reports that project partners are building up valuable experience of designing and developing innovative technology applications and that for many of them this is a novel activity requiring the acquisition of new knowledge and skills. Whilst projects’ reflections on their experiences of implementation suggests that a great many of them are rediscovering for themselves what is already known in the established systems development community, the importance of this learning should not be underestimated.
Our experience of R&D in many different sectors suggests that individuals and projects as collectivities need to progress through the various stages of the learning curve as a result of their own efforts and experiences, and that while it is possible to learn from what others have done this is not a substitute for more exploratory, trial and error learning in the unique context of one’s own innovative project.
Projects’ promotional activities
Projects’ have an image of themselves which is the basis for their promotional and awareness-raising activities. In general, projects define themselves in terms of a product or service which can be packaged or marketed with the various user populations. The longer term sustainability of some project offerings, however, may be more dependent on marketing a concept or an idea rather than a specific product which may well be overtaken by developments in technology.
Projects may therefore need to think more carefully about future promotional activities and to present themselves in terms of a ‘scenario of use’ rather than as a marketable commodity. Such a stance is consistent with developments in strategic marketing which have moved away from a focus on specific products to more integrated concepts conveying, for example, lifestyle choices.
User acceptance and usefulness
We have been struck by the efforts projects have made in obtaining feedback from users as part of a process of continuous improvement. Most commonly, feedback is elicited on user reactions to an early prototype or demo which is quite appropriate at this stage of the development cycle. In later stages, however, projects will need to concern themselves more with the wider question of the usefulness of the product or service, including how it will be incorporated into current work practices, social arrangements and organisational routines and behaviours.
Several projects have come to the realisation that the criterion of usefulness poses a much bigger question for evaluation than does user acceptance. Some have expressed concern that in cutting back on their evaluation they are not now well placed to look at the more important evaluative issues. We get a sense from the reports that whilst evaluation is one of the activities posing additional demands on projects, there is at the same time a new appreciation of its centrality to the innovation process.
Commercial partners as stakeholders
A noteworthy feature of eLib projects is the involvement of commercial partners, most notably publishers. As key stakeholders, however, they do not figure largely in project annual reports and it would appear that few, if any, project is systematically gathering data from the commercial sector on their perceptions, experiences or expectations.
The value of demos
Many projects have commented on the value of producing a rapid proto-type or demo for gathering feedback as part of the cycle of continuous improvement and as an effective means of raising awareness among the user population at the same time.
Evaluation methods
The diverse and multiple methods and techniques adopted by projects in their evaluations is impressive, particularly when we take account of the fact that many projects do not have experienced evaluators in the team but are developing their own evaluation capability among the partners.
We welcome the efforts of particular projects in developing and refining evaluation techniques that are suited to innovative, computer-mediated projects. These include both imaginative use of focus groups as well as the use of on-line data capture methods including questionnaires and usage data. The latter require further scrutiny as to their usefulness, but eLib projects can contribute to the development of protocols for recording and analysing data as well as exploring different strategies aimed at increased on-line questionnaire completion rates.
User-centred design
Recent thinking on systems/software development recognises its inherently social nature whereby innovative applications of technology are shaped by the interactions between the different actors involved. This necessitates an acknowledgement of their interests, agendas and powers as well as a means of reconciling or negotiating their differences, incompatibilities and even conflicts.
Several of the annual reports make reference to tensions within the project partnership, but these tend to be attributed to failures of communication rather than seen as more endemic features of a design process. What projects generally seem to lack is an appropriate set of mechanisms, procedures and roles that can support a dialectic communication process between the stakeholder groups as well as generating common understandings and negotiating differences. These might, for example, include a series of workshops, forums, brainstorming sessions and envisioning techniques.
We also observe extensive use of focus groups and other methods for eliciting feedback from users as part of continuous formative evaluation. There is only a small handful of projects, however, which appears to have given consideration to empowering the users in the development process. The most novel of these is the establishment of a Volunteer Group whose members receive training for their role and where an implicit social contract exists about their contribution to design decisions.
Finally, there is considerable scope, not currently exploited in eLib, for the development of joint strategies between individual projects and training and awareness projects in developing the knowledge and skills of the user target populations that would allow them to play a more active role as co-designers in the development process.
The Electronic Libraries Programme (eLib) was funded
by the Joint Information Systems Committee (JISC)
Page version: 1;
Web page maintained by
UKOLN Systems Team
and hosted by
UKOLN
- feedback to
systems@ukoln.ac.uk
.
Site last revised on: Tuesday, 24-Nov-1998 14:21:05 UTC
DC Metadata