|
Selection criteria for quality controlled information gateways
Work Package 3 of Telematics for Research project DESIRE (RE 1004) |
Title page
Table of Contents |
1 Scope Policy: Considering your Users
2 Content Criteria: Evaluating the Information
3 Form Criteria: Evaluating the Medium
4 Process Criteria: Evaluating the System
5 Collection Management Policy: Considering your Service
The scope policy of a service states what is and is not to be included in the catalogue. This is broadly defined at the inception of the service for the identified target audience.
In the selection process, the scope of the service will affect the first decisions made about the quality of the resources. Those falling outside the scope will be rejected, and those falling within it will go on through the rest of quality selection process, and be evaluated in the light of the rest of the quality criteria.
The scope criteria are the first filter through which the resources pass, and so are the most general criteria. They will tend to involve black and white decisions - either a resource falls within the scope or it does not. The most important thing to consider in choosing the scope criteria for a service will be the aims of the service and the target audience.
Information Coverage |
|
|
Subject Matter |
* What subject matter is appropriate for the target audience? * Are there any subjects which will be censored (e.g. for ethical reasons, such as resources produced by hate groups or resources about bomb-making/paedophilia) * How important is the subject matter of linked sites? |
|
Acceptable Types of Resource |
* What types of resource are appropriate for the target audience? * Is the information Scholarly rather than popular? * Does the resource contain more than just a list of links? * Is the site either proven to be, or expected to be durable? * Would a resource intended for use by an individual or local group be acceptable? * Is it innovative - does it make breakthrough design elements? |
|
Acceptable Sources |
* Which sources of information are acceptable/appropriate for the target audience? * Are academic, government, commercial, trade/industry, non-profit, private sources all acceptable? * Are pages maintained by individual enthusiasts (e.g. students) acceptable? |
|
Acceptable Levels of Difficulty |
* What level of resource is appropriate for the target audience? (e.g. users may be school children or may be academics) * Is biased information acceptable, and are opinions and ideologies acceptable? |
|
Advertising |
* Are resources that contain advertising acceptable? * Is there a limit to the amount of advertising that is acceptable? * Are there any forms of advertising which will be censored? |
Access |
|
|
Cost |
* How is charging going to affect selection - is the service only going to point to resources that are free to access? * Are there any price limits in terms of the access charge? * What if resources are under copyright? |
|
Technology |
* What technologies are appropriate for the target audience? (forms, ismaps, databases, cgi scripts, Java applications, frames, web sites, gopher, ftp, WAIS, telnet) * What connectivity does your audience have, and how will this affect selection? * What software do your users have and how will this affect selection? (E.g. will resources that work well in graphical-browsers but not in line-browsers be accepted?) * What hardware do your users have and how will this affect selection? |
|
Registration |
* Will the service accept resources where user-registration is necessary before the resource can be accessed? * Is on-line registration acceptable? * If users must negotiate written contracts before access is possible, is this acceptable? |
|
Security |
* When it is necessary for users to send confidential information out over the Internet, will the provision of a secure coding system or encryption affect the selection? |
|
Special Needs |
* Do your users have any special needs that will affect the resources selected? Large print or audio options for disabled users. |
Cataloguing Policy |
|
|
Granularity |
* At what level will resources be selected/catalogued? * Will resources be considered at the web site/Usenet group level or the web page/Usenet article level? |
|
Resource Description |
* What is the minimum amount of information needed to create a resource description in your catalogue? I.e. what basic information MUST a resource contain to be selected? (E.g. in a WWW document, contact details, last update details etc.) * Is there sufficient information to create a descriptive record (for your services 'minimum set') |
|
Metadata |
* Will the service accept resources with/without specific metadata? |
Geographical Issues |
|
|
Geographical Restraints |
* Are any geographical restraints appropriate for your audience? * Will the service cover information produced locally, from particular countries, particular continents or world wide? |
|
Language |
* Resources in which languages are acceptable/appropriate for your target audience? |
Currency and Adequacy of Maintenance |
|
|
(See 'Resource Integrity' section for details of this section) |
|
These criteria are based on the information content of resources. The fact that the resources are Internet resources is not so relevant to the criteria in this section, indeed many of these criteria have been used by librarians in the selection of books and traditional information resources for many years.
The criteria are listed in the main headings and the left-hand column. The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.
Validity |
|
|
How valid is the content of the information?
|
* Are references given? * Is there a bibliography? * Does the resource have a scope statement? * Is there any information missing? * Is there any mention of the resource being available in another format? |
|
Is the information what it appears to be?
|
* Is it merely vanity publishing? * Is there a request for payment? * The URL - Does the URL support the claim of authorship? * Email addresses given - Are emails for a publisher, the author, referees, sources, etc. given? * Contact details given - Are addresses and phone numbers given that support claims of authorship, sponsorship etc.? * Traffic levels - Are they high? |
|
Is the content of the resource verifiable - can you cross check the information? |
* Are the sources of the information stated? * Is it an often-cited source? |
Authority and Reputation of the Source |
|
|
Who provided the information?
|
* Is the origin of the document documented? * Author's title and institution/company displayed? * Is the information attributed to an author or editor? * Have you heard of the author before? * Is the URL a university server? * Is the author someone who has been cited frequently by respectable sources? * Do you know the educational background of the author? * Do you know the occupational background of the author? * Is the information written on a topic in the author's area? * Is the author's full name displayed? * Has the author had previous publications of note? * Is a biography of the author given? |
|
Has the information been filtered? |
* Is the site sponsored by a company, organisation or individual widely recognised as an authority or expert in the field? * Is the information peer-reviewed? * Has it been refereed? * Is the site sponsored? * Has the site been reviewed by a content reviewing agency? * What is the level of moderation for mailing lists and Usenet newsgroups * Via what source did you come across the resource? (i.e. did someone authoritative recommend it? * Is there a common link to the page from a recognised authority? * Is the site linked to by multiple Internet sites?
|
|
How reputable are the 'filters'? |
* Has the material been disseminated by a trade publisher? * Is the publisher known to you and reputable? * Is the publisher a recognised authority? * Is the publisher a 'University Press'? |
|
Is it by who it says it's by? Can the authorship be validated? Can claims to having been 'filtered' be validated? |
* Can information located in the publication itself be used to determine the author's credentials? * Is the author listed on say, the campus directory/organisational directory? * Are there email contacts for the publishers/referees/sponsors? |
Substantiveness |
|
|
Is the information substantive? |
* Does the resource contain more than contact details? * Is the information full-text? (As opposed to just titles/bibliographic details) * Is it merely advertising? |
|
Is there value-added information? |
* If the resource consists of a collection of links is there substantial annotation or value-added information? (e.g. an annotated bibliography) |
Accuracy |
|
|
Is the information accurate? Is the accuracy of the presentation adequate? |
* Are you able to check the accuracy of the information? * Does the page cite a bibliography or provide references to confirm the accuracy of the information? * Is the grammar and spelling accurate? * Is there a prevalence of typographical errors? |
Comprehensiveness |
|
|
What is the depth of the information? To what level of detail does the resource go? How superficial/exhaustive is the information? Is some of the information incomplete? What is the breadth of the information? Are all aspects of the subject covered Is everything you expect to find in the site there? Are there any logical gaps of information? |
* Is the title informative? * Is an abstract given? * Is there an opening mission statement of the purpose of the resource? * Are there stated criteria for inclusion of information? * Does the index or contents page imply comprehensive coverage? * Are key words given that indicate the information content? |
Uniqueness |
|
|
Is the information on the site unique? |
* Is it primary material? * Is there any original work available at the site? * Does the material have any relation to other works? * Is the site inward focused i.e. not just a list of links to external sites? * Is there any value added? |
Composition and Organisation |
|
|
Is the information composed well? Is the information clearly organised? |
* Does the text follow basic rules of grammar, spelling and literary composition? * Does it include jargon? * Is the information within a resource phrased unambiguously? * Is there a good structure? * Is the information within a resource arranged logically and consistently? * Is the information broken down into logical parts? * Is the resource well laid out? * Is the resource organised by the needs of the user? Is the information broken down into digestible parts? * Is the content clearly described? * Are the headings clear and descriptive? * Is there evidence of internal standardisation (e.g. use of a 'style sheet'?) |
Currency and Adequacy of Maintenance |
|
|
(See 'Resource Integrity' section for details of this section) |
|
Form criteria are based on the presentation and organisation of the information. Some of these criteria would be as applicable to paper resources as to Internet resources. Users need to be able to find their way around information objects, however, the evaluation of Internet objects will involve checking electronic forms and so will involve different evaluative processes
Once again the criteria are listed in the main headings and the left hand column. The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.
Ease of Navigation |
|
|
Is it easy to navigate the resource? Is it easy to browse the resource? Is it easy to search the resource? |
* Are there hidden layers that are difficult to discover? * Does it take more than three 'clicks' (three links) to get to something interesting? * Do all the links serve an easily identified purpose? * Are all the links clearly labelled? * Do you ever find yourself in a position where there are no hyper-links to anywhere else? * Are hyper-links ambiguous i.e. is it obvious where a link is leading you to? * Do images support ease of navigation? * Are graphics/sounds/videos clearly labelled and identified? * Can pages or portions of a document be printed separately? * Are there single document options for those resources that may be printed? * For discussion in mailing lists and Usenet groups, are digests available? * Is there an index? * Is the resource indexed electronically? * Can a particular page be located from any other page? * Are there good back and forward links between pages? * Are the individual web pages concise or do you have to scroll forever? * Does the system have an effective search facility? * Is keyword searching possible? * How effectively can information be retrieved from the resource? * Is a well know search engine provided? * Does the search engine allow the use of Boolean operators? |
Provision of User Support |
|
|
Are there instructions? Is there documentation? Is there online Help? Is customer support and training provided? |
* Do essential instructions appear before links and interactive portions? * Is there online documentation? * Is print documentation available? * Is there any online help? * Is contextual help available? * Is there an email 'Help Desk'? * Is there a telephone helpline? * Are training materials/courses provided? |
Use of Recognised Standards |
|
|
Are recognised standards used? |
* Is metadata provided? * Does it use standard multimedia formats? (e.g. MIME) * Is it written in standard HTML? * Have proprietary extensions to the HTML been added that some browsers will not recognise? |
Appropriate use of Technology |
|
|
How appropriate is the format? |
* Does it do more than can be done with print? * Is appropriate interactivity available? |
Aesthetics |
|
|
Has consideration been given to the appearance of the site? Does the resource follow good design principles? |
* Does it look and feel friendly? * Is the balance of text, images, links, headers, font sizes and white space good? * Are the size, colour and animation of the images appropriate? |
Information Integrity (work of the Information Provider) |
|
|
Is the information current and up to date? Is the information durable in nature? Is there adequate maintenance of the information content? |
* If the site contains data or information that is time-sensitive, how current is this data and information? * How current is the material included in each update? * Is a date given stating when the web item was mounted? * Are time-sensitive resources available in near real-time? * Do the stated dates respond to the information in the resource? * Is the date given stating when the web item was created? * How time-sensitive is the information, and how does this relate to frequency of update? (e.g. for resources such as timetables, schedules and conference announcements) * If it is a static resource (not updated) will the information be of lasting use to the audience? * Is the information of a type that has a limited period of use? * Is the information provider likely to be able to maintain the information (unlikely in the case of information provided by students). * Is the resource improved and enlarged and updated appropriately? * Has the data been updated recently? * Is there a statement about the frequency of update? |
Site Integrity (work of the Web-Master/Site Manager) |
|
|
Is the site current and up to date? Is the site either proven to be, or expected to be durable? Is the site adequately administered and maintained? |
* Are there any dead links? * Are all the pages dated with the last revision date? * Are there links to sites that have moved? * Is a version number for the resource displayed? * Date of last update to the resource displayed? * Is there a description of the update frequencies for the resources? * Are you being redirected to a new URL? * Does the organisation or person hosting the resource seem to have the commitment to the ongoing maintenance and stability of the resource? * Is the site frequently updated/maintained? * Is the site regularly updated? * Are the downtimes announced? |
System Integrity (work of the Systems Administrator) |
|
|
Is the technical performance of the resource acceptable? Is the system stable? Are adequate measures taken to maintain the integrity of the system? |
* Is the resource currently accessible? * Are the connections to the site providing the information reliable and stable? * Is it usually possible to reach the site or is it overloaded? * Are the downtimes infrequent? * Are the links reasonably stable? * Can you review the peak usage time for the resource (to assess reliability)? * Is the site mirrored? |
Process criteria are based on the processes that exist to support the resource. Unlike form and content criteria, these will be related closely to the fact that these are Internet resources. The fact that information on the Internet lacks the integrity of a published work raises a number of questions about the quality of a resource over time. The system that lies between information provision and information retrieval contains many variables, and some evaluation of these is needed to discern the quality of the resource.
The criteria are listed in the main headings and the left-hand column. The right hand column contains some hints and checks that might be used to discern whether a resource meets with a particular criterion.
Collection Coverage and Balance |
|
|
What's already in the collection? What is the relative value of the resource in comparison with others already in the collection? Is the information unique within the context of the total collection or does it duplicate? |
* Do a search to avoid duplication * Browse to see which areas are well covered/where gaps in the collection exist |
Availability of Internet Resources |
|
|
What's available outside the collection? What is the relative value or the resource in comparison with others available on this topic? Is there similar/better subject material available? Does the site contain information that is not readily available: would you have to look long and hard to find similar data? |
* Look for other resources with value added to the information |
Availability of Library Resources |
|
|
What's available via other information sources? Does the resource provide access to information that is not easily obtained in other formats/not held in many libraries? Does the resource provide access to information that users may not be able to find/get access to in libraries? |
* Look for resources that compliment existing information/library collections. * Look for electronic versions of texts that are in great demand in libraries in paper format, and therefore, difficult to get hold of. |
| Next | Table of Contents |
Page maintained by: UKOLN Metadata Group,
Last updated: 5-Feb-1999