Previous - Next

PART II - 4

PART III - between 1 and 2

Appendices - between 4 and 5

PART III - 3

Appendices - 5

by shifting repetitive details of the methodology and procedure to an

additional appendix. Also, conclusions need to be emphasised (but not reiterated) more clearly.

part, nested hierarchical structure, a scheme which NOT followed by the subsequent Content Criteria Form Criteria Process Criteria

Collection Management Criteria sections.

sheer number of evaluative questions (127) and sub-questions

(additional 127 items) makes it a precise, informed but clumsy and

labour-intensive tool.

(a) a study of the intra-evaluator consistency (does the same person

give similar reply to the same question referring to the same resource seen for the 2nd or 3rd time ?)

(b) a study of inter-evaluator consistency (does the model yield

similar results for the same set of resources if used by different

evaluators ?)

(c) a study of the advantages/disadvantages of the proposed model as

compared with other evaluative schemes. Certainly, only a demonstrably better tool should be used.

Reliance on human selection has its strengths and weaknesses and it is acknowledged that for any evaluator the stringency of application of any set of criteria is variable.

We need to make sure that there is an acceptable level of reliability and consistency in the decisions made. Decisions made about the SCOPE criteria are likely to be close to 100% consistent, as these criteria and black and white. Decisions based on the other criteria are likely to be less reliable as the evaluation involved is more complex. I think further work is not viable in DESIRE 1

KB, Lund and SOSIG selection staff will be working on distilling the model to create agreed working criteria for the three gateways this Summer - and hope to highlight the most important criteria which should improve the consistency of decisions. This will be a necessary precursor to any future development work on weighting and scoring. (envisaged within the DESIRE II proposal).

We have now established discussion lists for European Section Editors and Correspondents, which provide a forum for reaching consensus on issues around the consistency of the selection processes in general and where appropriate individual resource selection.

When individual services distil the model, and have a more compact list of agreed working criteria with particular emphasis, then intra-and inter-consistency will improve.

The model allows for formal review of the selection criteria employed, but in reality this should be achieved by open discussions about the selection criteria of a service. People can continually compare and contrast their decisions and reach and agreed understanding of the criteria. This is now happening via the discussion list which we have established.

Currently we do not have resources to undertake such a study, though an informal comparison with a set of selection criteria developed by Alison Cooke ( as part of her PhD at Aberystwyth University) will be made.

Rowland Hill Street, London NW3 2PF, United Kingdom

As OMNI Assurance Officer, Betsy produced OMNI's Guidelines for Volunteers: Evaluating Resources for OMNI, and has continued to develop and adapt these as necessary. She also convenes the OMNI Advisory Group for Evaluation Criteria, which has been assessing the use of evaluation criteria by Internet review sites and evaluative subject gateways. Betsy has written evaluative reviews of electronic products and publications for Information World Review and Managing Information, and has given a number of presentations on issues of quality of information.

She has authored or co-authored the following papers in the area of networked information quality (most recently reporting work of the OMNI Advisory Group for Evaluation Criteria):

Anagnostelis, Betsy. Filtering services and quality of information on the Internet, Inform 180, December 1995. p3.

Anagnostelis, Betsy and Cox, John. Data on the Internet: Evaluating the Quality or "Less is More". In C. J. Armstrong and R. J. Hartley (eds.)

ukolug96@warwick.ac.uk: UKOLUG State-of-the-Art Conference, Warwick, 17-19

July 1996, London: UKOLUG, 1996. pp59-69.

Cooke, Alison, McNab, Alison and Anagnostelis, Betsy. The good, the bad and the ugly: Internet review sites. In Online Information 96: the Proceedings of the 20th International Online Information Meeting, London,

3-5 December 1996, Oxford: Learned Information: pp33-40.

Anagnostelis, Betsy, Cooke, Alison and McNab, Alison. Thinking critically about information on the Web. Vine (forthcoming).

Subject gateways consciously emphasise the importance of skilled human involvement in the assessment and 'quality control' of their selected Internet resources. The core activity - selecting and attributing meaning to those resources is a human activity. Subject gateways are run as academic services.

Not exclusively academic services? It is conceivable that

non-academically run SBIGs might develop?

...

Root Definitions

A university owned and maintained system that selects and catalogues subject specialist Internet resources on the bases of quality and relevance, allowing structured access by a range of users in research and education...

What about practitioners?

...

The language used was standardised. The criteria and attributes were phrased as a question. The question format was chosen because it was the most common format found during the review, and because it reflects the evaluative nature of the selection process.

Some clear definitions could additionally have been provided?

Nonetheless, I'm happier that:

It was therefore decided that none of the items should be removed from the list, as they might be appropriate for some services.

Weighting can possibly turn out to be a key concept when it comes to distinguishing which criteria apply to different disciplines. This, again, might be an area for further exploration.

Might copyright warrant a mention as a separate category? It's not strictly speaking cost related

Granularity

At what level will resources be selected/catalogued?

Will resources be considered at the web site/Usenet group level or the web page/Usenet article level?

This defines almost how the questions might best be phrased, and their relevance (in the document, there is a preponderance of questions referring to "sites"); in the long term, it might be useful to distinguish among criteria that apply more to sites than to individual resources or collections of resources.

What is the minimum amount of information needed to create a resource description in your catalogue? I.e. what basic information MUST a resource contain to be selected? (E.g. in a WWW document, contact details, last update details etc.)

Perhaps mention authorship as an example? As an illustrative example, it would be particularly apposite.

Who provided the information?

Is the source attributable to a reputable author or organisation?

Is the URL a university server?

or other reputable organisation?

Is the site linked to by multiple Internet sites?

By other selective subject gateways? (Perhaps some discrimination of the quality of the linking site might be a good idea: not all services that might be considered to provide selectivity are perhaps of equal standing?)

Perhaps worth making a specific mention of mirror sites? As a special case, maybe. Worth mentioning them in the scope notes, too? (Are they acceptable?)

I would agree that Currency is a feature of a resource, and should be

accommodated in this section. I would argue that Adequacy of Maintenance is information provider related and can slip out of this section and fit more appropriately in the 'Resource Integrity' section. Perhaps the best solution is to separate the two concepts entirely?

Is it easy to navigate the resource?

Does it take more than three 'clicks' (three links) to get to something

interesting?

To get to 'substantive information'? ('Something interesting' is very

subjective; admittedly, 'substantive information' might be too restrictive. Perhaps define it as something 'relevant to the objective' of the service?)

Does the system have an effective search facility?

...

How effectively can information be retrieved from the resource?

I'm not sure I understand the difference between these two questions?

Is the information current and up to date?

If the site contains data or information that is time-sensitive, how current is this data and information?

How current is the material included in each update?

Is a date given stating when the web item was mounted?

Are time-sensitive resources available in near real-time?

Do the stated dates respond to the information in the resource?

Is the date given stating when the web item was created?

Is the information durable in nature?

How time-sensitive is the information, and how does this relate to frequency of update? (e.g. for resources such as timetables, schedules and conference announcements)

If it is a static resource (not updated) will the information be of lasting use to the audience?

Is the information of a type that has a limited period of use?

These two paragraphs could make up a new 'Currency' section under Content Criteria.

This is a fascinating review of quality models, unravelling interesting possibilities. I'm unclear about how the evaluation criteria relate to this specifically, although as a broad outline I have no difficulty with it.

1 Introduction

Nineteen selective subject gateways

I have a little bit of difficulty with the use of the same descriptive

phrase 'selective subject gateways' in this latter context (Appendix V:) -

or else with the definition of a selective subject gateway as a university-based service primarily. Some critical work has been conducted in comparing the various types of services; I hesitate to suggest the publications, as I am co-author of all three (one forthcoming).

Conclusions

As noted in the introduction most of the surveys carried out were not designed to elicit feedback related to resource selection criteria...

While mentioned later on in Appendix VIII, this might also be an

appropriate place to mention Alison Cooke's work with OMNI. (Mentioned under: End-user perceptions of information quality in an network environment

One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself. One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.) In fact a correction is due here, since Alison is conducting research in her own capacity - OMNI is simply providing the test bed.

Granularity

...The key question is: at what level should selection take place?

Individual Files? The Server? This may not seem to be particularly relevant to quality selection issues, but quality guidelines may be different for different levels".

Yes - these match my comments above under Granularity. Interesting that they have been picked up through the email questionnaire...

formalise criteria according to different types of resources (e.g.

electronic journals versus image databases, etc.)".

Alison Cooke is now beginning to conduct research specifically into this aspect.

Several subject gateways noted the need for increased co-operation with the information providers themselves. Some services already contact information providers to fill-in gaps in their knowledge about a resource (if for example a resource was missing a date) and would be prepared to contact them again to check whether resources were going to be updated regularly. Others would like information providers to contact them with details of new or updated sites when necessary or to make some comments on the resource description made by the service.

Indeed, this is something I would add to the model: draw an arrow from 'add to catalogue' to 'request/inform provider'?

One respondent noted that criteria currently used were not validated with reference to their suitability for the network environment or to the subject areas being covered by the database itself. One of the subject gateways (OMNI) is conducting research into end user perceptions of information quality.

See comments above about desirable correction here.

It is acknowledged that creating lists of criteria that are specific to a particular category of networked resources (server/document/mail archive etc.) could be usefully developed. (Indeed the reviewer points out later, that useful work is being done in this area by Alison Cooke of Aberystwyth University).

Agreed that in the final criteria list this will be changed to: 'Has the information been through a reputable filter?'

Is the site linked to by multiple Internet sites....

Agreed. In the final criteria list this will be changed to: 'Is the site linked to by any reputable sites?'