QA Focus Final Report
[ Home Page - Documents - Presentations - Search ]
This is the final report for the QA Focus project.
Please note that the master version of this report is available in MS Word format.
| Version | Date | Comments |
|---|---|---|
| 1.0 | 2 Nov 2004 | Updated version, following suggestions received from JISC. |
| 0.9 | 28 Oct 2004 | Version submitted to JISC at meeting on 29 Oct 2004 |
Note the name of the JISC programme, and that the project was funded by JISC. The project may also want to list the project partners and acknowledge any person or organisation that was helpful during the project or in writing the report.
QA Focus was funded by the JISC 5/99 Learning and Teaching and Infrastructure programmes (see <http://www.jisc.ac.uk/landt/>) from January 2002 to December 2003. It was then extended from January-July 2004.
From January-December 2002 QA Focus was provided by UKOLN and ILRT. ILRT withdrew from the project at the end of the first year and were replaced by AHDS who were partners with UKOLN from January 2003 to July 2004.
The QA Focus team members during the lifetime of the project have been: Brian Kelly, UKOLN (January 2002 to July 2004), Marieke Guy, UKOLN (January 2002 to December 2003), Amanda Closier, UKOLN (February to July 2004), Karla Youngs, ILRT (January-December 2002), Ed Bremner, ILRT (January-December 2002), Hamish James, AHDS (January 2003 to July 2004) and Gareth Knight, AHDS (May 2003 to July 2004).
Our initial contact at the JISC was Caroline Ingram. Following Caroline's departure our contacts at JISC were Rachel Bruce and Balviar Notay.
Summarise highlights of the project (one page), including aims/objectives, overall approach, findings, achievements, and conclusions. The full report may include technical terms, but try to keep the executive summary in plain English.
The aim of the QA Focus project was to develop a quality assurance (QA) methodology which would help to ensure that projects funded by JISC digital library programmes were functional, widely accessible and interoperable; to provide support materials to accompany the QA framework and to help to embed the QA methodology in projects' working practices.
The approach taken was initially to liaise with selected projects in order to gain an understanding of the broad areas in which difficulties in ensuring project deliverables would be interoperable and to solicit feedback on solutions based on a quality assurance approach. These focus groups were supported by a number of automated surveys of project Web sites which sought to gain an understanding of problem areas related to the provision of project Web sites. We learnt that the provision of technical support and advice would be appreciated by projects, but there was a concern over the development of too bureaucratic and time-consuming QA methodology.
As a result of the initial liasing with projects, we supported projects using a three tier approach:
Initial guidance was provided the form of brief, focussed briefing documents that promoted (a) a standards-based approach and, more specifically, the significance of open standards; (b) Appropriate implementation architectures for deploying standards and best practices and (c) approaches to testing to ensure that standards and best practices were being implemented correctly.
Secondly, we commissioned case studies to enable projects to describe the approach (or approaches) they had taken, the standards they had used, and the practices they had implemented or developed in their own words. To ensure the documents contained practical lessons and not marketing materials for the project, the case studies template required authors to describe not only their successes, but also the lessons they had learnt and the things they would do differently. The case studies helped to foster a spirit of sharing across the programme.
In addition to publishing a range of documents for use by the projects we developed a quality assurance framework. The quality assurance framework was built from the documented policies outlined in briefing papers and case studies, together with use of systematic procedures for ensuring the policies were being implemented correctly. Conscious that a heavyweight bureaucratic solution would restrict or be ignored by projects, we sought to develop a lightweight approach which would be easily understood and implemented, provide clear benefits to the projects in ensuring their deliverables were functional, and help to ensure deliverables were interoperable with others. Using this material, projects were able to perform a self assessment process to examine their procedures and gain immediate feedback.
In parallel with the creation of a QA framework and support materials on the project web site we sought to validate our approach by submitting papers to a number of peer-reviewed conferences. Papers that outlined our ideas and approach were accepted by a number of peer-reviewed conferences. These were published in the EUNIS 2003 Conference Proceedings (later republished as one of the best conference papers in the Informatica journal); on the implementation of our approaches by others published in the IADIS Internet/WWW 2003 Conference Proceedings; on approaches to the selection of open standards in the ichim03 Conference Proceedings and on the use of the methodology by other digital library programmes published in the ECD 2004 Conference Proceedings. Papers on our work will also be published shortly in the Canadian Journal of Learning and Technology (on the application of a QA approach to e-learning accessibility) and Library Trends (on QA for metadata).
The approaches taken by the QA Focus project are currently being adopted by new JISC programmes and have been included in JISC's Project Management Guidelines. To allow continued development and ensure wide impact of the project output we intend to licence the 70+ QA Focus briefing papers available on the web site under the Creative Commons.
Summarise the background to the project (and how it builds on previous work) and the need it for it (and why it's important).
The development programmes funded by JISC encourage the use of open standards to ensure interoperability. Although advice was provided on appropriate open standards, initially in the eLib Standards document and later by its successor, the DNER Technical Standards document, there has not been a formal checking procedure that ensured open standards were used by projects in practice. This approach may potentially lead to problems and limit the capabilities of funded projects.
Concern for these issues resulted in a call being issued by the JISC to explore the potential for a quality assurance framework which could help to ensure that project deliverables were interoperable and to develop a QA methodology for use by projects. The Digitisation and QA Focus, as it was initially called, was recognised by JISC as essential to ensure QA methodology was usable on a practical level within the development environment of projects.
The initial remit of QA Focus's work was to provide support for JISC's 5/99 Learning and Teaching and Infrastructure programmes (although the scope was later extended to cover other JISC development programmes).
List the aim and objectives agreed at the start of the project, and note if they changed during the project.
The aims of the QA Focus project were to:
Summarise the overall approach taken and why this approach was chosen over other options considered. Then describe the methodology in more detail. Depending on the project, this might include the methodology for research you carried out, technical design or development, evaluation, etc. Finally, note any specific issues that had to be addressed by the methodology, e.g. standards, interoperability, scalability, etc.
The QA Focus team initially announced their aims at a JISC 5/99 all-programme meeting shortly after funding for QA Focus had been granted. This was followed by discussions held at focus group meetings, hosted by MIMAS and EDINA that enabled two of the key JISC Services together with JISC 5/99 projects in the two regions to provide their views on areas such as the JISC DNER standards document itself, issues concerning selection of standards, development and deployment issues. In parallel with this work we carried out a number of surveys of project Web sites in order to identify any shared problem areas across the projects in complying with Web standards and to identify examples of best practices.
These initial activities helped us to recognise key themes among the sampled projects
Based on this initial feedback the QA Focus team identified a number of strands to its work:
Details of how we implemented these strands is given in the following section.
Describe how you planned and implemented the project work and the activities it involved. Depending on the project, this might cover technical development, processes, how you conducted user studies, etc. Include any problems or issues that arose and how you handled them, where readers can learn from your experience. Tell the story of what you did rather than listing workpackages.
Planning for the QA Focus project took place over several weeks, while team members contacted staff at several projects to identify the quality assurance requirements within their field of expertise. Based upon these interviews (conducted through e-mail and face-to-face), the QA Focus team were able to establish methods of educating projects on established standards and gain a better understanding of existing practices. The geographical proximity of team members at UKOLN and ILRT (Bath and Bristol respectively) allowed frequent face-to-face meetings to establish a QA approach to digitisation. This was complemented by managed use of collaborative technologies (mailing lists, instant messaging and shared file store) to liase.
Towards the end of the first year, ILRT announced that they were withdrawing from the project in order to re-focus on core activities. Fortunately we were able to replace ILRT by the AHDS, with whom UKOLN was involved in other joint project work. Once the AHDS had appointed staff to work on the QA Focus project, we were able to build on our initial work. A workplan was created that outlined the work to be completed and the deadline for completion.
Since AHDS were based in London, it was felt that regular face-to-face meetings would be time-consuming and expensive. Although we held a number of face-to-face meetings we placed greater emphasis on use of collaborative technologies, focussing upon the QA Focus mailing list and instant messaging service for personal and group intercommunication. This worked well and allowed communication between team members, irrespective of their location at the time.
During the latter half of 2003, the QA Focus officer at UKOLN went on maternity leave for a period of 6 months. It was felt that, due to the difficulties in recruiting staff for such short periods of time, we would seek to continue the QA Focus work from existing staff, but at a reduced level. One of the consequences of this was that it was not possible to be as involved in face-to-face meetings with the JISC 5/99 Teaching and Learning and Infrastructure project holders to the level originally envisaged.
Despite these external factors, QA Focus succeeded in developing a QA framework and support materials during its second year, together with a number of papers and presentations at peer-reviewed conferences which helped to validate our approach. Towards the end of the second year we outlined the QA methodology and the support materials at a JISC meeting. Our work appeared to be appreciated, and JISC programme managers from JISC programmes we had not been involved with expressed their interest in QA Focus supporting their programmes. Following agreement from the JISC, we extended our remit from the 5/99 programme to the FAIR and X4L and other JISC programmes. We subsequently provided support fore the FAIR, X4L and Digitisation programmes.
Explain the end result of the project work in an objective way. Depending on the project, it might include research results, findings, evaluation results, data, etc. If the project created something tangible like content, a portal, or software, describe it. Engage the reader, and avoid a long list of deliverables.
The project outputs were:
The QA framework is a lightweight framework, based on the provision of technical policies together with systematic procedures for measuring compliance with the policies. The QA Framework is described in a number of the QA Focus briefing documents and the rational for the framework has formed the basis of a number of peer-reviewed papers
The briefing papers are short, focused documents that cover the need for particular standards, the advantages and disadvantages of particular implementation architectures, examples of common problems and how to avoid them and approaches for checking compliance with standards and best practices. The documents have designed to be modular, enabling them to be used in a variety of ways and are provided in several formats to maximise their usage. The contents of the briefing papers will be made available under a Creative Commons licence which is aimed at maximising the impact of the QA Focus work and helping to embed the approaches more widely. Documents are provided in XHTML for online reading and MS Word format for printing of an A4 and/or double-sided A5 printing.
The case studies provide an opportunity for sharing of experiences and best practices across the community. The case studies reflect the experiences of the projects and any difficulties that have been experienced.
The toolkits provide a structured, checklist approach to help projects ensure they have addressed key issues. The toolkits are available as paper-based checklists and are complemented by online forms which provide a more interactive interface. The toolkits can be used in a number of ways such as providing a checklist to support decision-making by project developers or project managers; for use by project partners or for use in a workshop environment. For example the toolkit has been used to support a workshop on QA for JISC's Digitisation programme.
The QA Handbooks are intended to allow users to download the QA Focus materials in a format suitable for printing.
The peer-reviewed papers provide the rationale for various aspects of the QA Focus work.
In this section, assess the value of the project work. List project achievements against the aims and objectives set. Summarise project outcomes and their impact on the teaching, learning, or research communities. Indicate who will benefit from the work, how, and why. Also comment on what you learned that may be applicable to other projects, e.g. whether the methodology worked.
The QA Focus project has successfully developed a QA framework and support materials and advice and documentation on use of the QA framework.
The QA framework should be relevant for all JISC programmes and not just those initially targeted.
Briefly summarise any conclusions that can be drawn from the project work.
The main conclusions from the QA Focus work are:
Consider the future implications of your work and how others can build on it. What are the implications for other professionals in the field, for users, or for the community? What new development work could be undertaken to build on your work or carry it further?
The Garner curve of new technologies describes how assessment of technology changes over time. In the initial stage technology is used by early adopters and the benefits are promoted, resulting in hype and unrealistic expectations. These unfulfilled expectations can lead to a 'trough of despondency' when the technology fails to live up to the hype and costs begin to escalate. Afterwards a more realistic appreciation of the role of the technology is gained and the technology becomes deployed in a service context (the 'service plateau' in Gartner's terminology). This curve and its application to technologies used in digital library programmes is illustrated below.
The move towards service deployment is often accompanied by a more managed approach to use of the technology, including deployment of QA techniques. We can see that in mature areas of technology, such as software development and digitisation, QA procedures are already well-established (even if they are not necessarily widely implemented within the community) and that QA for Web development is currently being developed (e.g. work in this area by QA Focus). However in areas such as the use of metadata technologies (e.g. OAI, Dublin Core, etc.), use of Web services technologies (SOAP, REST, WSDL, UDDI, etc.) and Semantic Web (RDF, OWL, etc.) there is only limited agreement on the underlying technologies which mean that QA procedures related to these areas are non-existent or are in the early stages of development.
Since metadata and web services will form the basis of JISC's Information Environment, we feel there is a need to begin developing a QA infrastructure, with the aim of minimising the effects encountered in the 'trough of despondency' and assisting in the transition to a service plateau.
The QA framework has a wider applicability outside of the JISC digital library development area, including use within the museums, libraries and archives sector, use by other digital library programmes (e.g. in Europe and the US) and use to support best practices for providing conventional Web sites (e.g. by UK HEIs, further educational colleges, by funding bodies, JISC services, etc.
The QA framework has also being adopted as part of a holistic approach to e-learning accessibility which has been developed by TechDis and UKOLN. This model has been accepted for publication in a special issue of the Canadian Journal of Learning and Technology.
The work of the QA Focus team has direct relevance to the approaches taken by JISC to use of standards by JISC funded projects.
There is a need for JISC to clarify the scope, change control and expectations of compliance with the JISC standards document. The scope of the document could be just formal project deliverables through to providing a resource which could be used by the wider community; the document could mandate the standards to be used, with penalties for non-compliance or could describe best practices which JISC would encourage projects to use, but would accept non-compliance.
Although use of open standards is important to help ensure interoperability, selection of open standards is not a guarantee of success. Open standards may be immature, costly to implement, may change in light of deployment experiences or not be accepted by the marketplace. The JISC community has experiences of the failures of open standards with OSI networking protocols and coloured book software in the 1980s. However leaving projects to do their own thing is likely to lead to interoperability difficulties. There is therefore a need for an alternative approach, such as the selection matrix for open standards which QA Focus has developed for use in conjunction with policies determined by the funding body e.g. JISC could mandate certain standards by particular programmes (such as use of OAI-PMH for use by the FAIR programme which seeks to gain experiences of this protocol) but leave other decisions to the projects themselves, possibly supported by reporting on deviations or requiring approval by JISC programme managers, project steering groups, etc..
There is also a need to clarify the scope, change control and expectations of compliance with the JISC standards document. The scope of the document could be just formal project deliverables through to providing a resource which could be used by the wider community; the document could mandate the standards to be used, with penalties for non-compliance or could describe best practices which JISC would encourage projects to use, but would accept non-compliance.
List any specific recommendations for the teaching, learning, or research communities.
Based on our work the QA Focus team has made the recommendations which are listed below. It should be noted that the project team informed the JISC of its recommendations during the life of the project and, consequently, a number of the recommendations have already been implemented
List any references to the work of others you have cited (e.g. articles, reports, studies, standards), and any explanatory notes. Provide URLs for any materials available on the web.
The QA Focus QA Methodology is described at <http://www.ukoln.ac.uk/qa-focus/documents/case-studies/>.
QA Focus briefing documents are available from <http://www.ukoln.ac.uk/qa-focus/documents/briefings/>.
QA Focus case studies are available from <http://www.ukoln.ac.uk/qa-focus/documents/case-studies/>.
QA Focus peer-reviewed papers are available from <http://www.ukoln.ac.uk/qa-focus/documents/papers/>.
QA Focus handbooks are available from <http://www.ukoln.ac.uk/qa-focus/documents/handbook/>.
QA Focus toolkits are available from <http://www.ukoln.ac.uk/qa-focus/toolkit/>.
Brian Kelly, QA Focus project manager, UKOLN
2 November 2004
[ Home Page - Documents - Presentations - Search ]
Web page by Brian Kelly of UKOLN.
Last Modified 29-May-2005
Email comments to webmaster@ukoln.ac.uk