Assessing Learning Materials from JISC-funded Resource Databases

Background

The aim of the Healthier Nation [1] project is to identify suitable learning materials from JISC-funded resource databases and/or content repositories and re-purpose a sample of the material as learning chunks to support Health and Social Care curriculum delivery (at FE and HE Levels. The project has specifically focused on "the Big Four" diseases affecting Scotland: Cancer, Coronary Heart Disease, Stroke and Mental Illness.

Problem Being Addressed

The first Strand of the project has been concerned with research and evaluation. The objectives of strand 1 have been to identify and evaluate relevant materials on the "big 4" diseases by:

• Carrying out an electronic search of defined and agreed databases and content repositories
• Evaluating the resources for appropriateness to Health Care related programmes at Scottish Credit and Qualifications Framework (SCQF) levels 4 to 7 [2]
• Identifying keywords and signposting information (levels, courses, delivery options) to facilitate the development of metadata
• Highlighting areas where little or no learning materials exists to support learning in the identified subject areas at the appropriate levels.

The Approach Taken

Research Methodology

Academic experts from each partner institution prepared a 'mapping grid' to assist the research team to identify relevant resources. The grid included information on key subject areas, specific keywords and exclusions, courses which could use the material (including the level) and any relevant learning outcomes.

As the emphasis for the academic staff was on finding resources that could be used in teaching situations at FE and initial HE levels - possibly with some element of re-purposing - the research team concentrated on the relevant subject gateways, rather than bibliographic sources and indexes.

To provide a structured framework for the evaluation of the learning material, resource sheets were used to record relevant details. The sheets ensured that the evaluation criteria used by all partner institutions were consistent.

The researcher team evaluated all the learning material on their content (clarity, authority, bias, level) and their style and functionality. Copyright details were also recorded for future re-purposing. Restricted vocabularies were used whenever possible to assist metadata tagging of learning objects. Resource sheets were then passed to academic staff to evaluate their appropriateness for teaching and to indicate how they could be used (delivery type, mapped to level/course, any re-purposing that would be required).

The intention was to carry out accessibility evaluations on a selection of the resources during this part of the project. A key issue that has affected this work has been the lack of agreed criteria for accessibility evaluations. One of the project partners, RNC, has been working with TechDis to develop a model for evaluating resources and the accessibility evaluation of the resources will now be carried out at a later stage of the project.

Following evaluation, the materials were:

• Included in a final list of resources to be uploaded to a repository
• Included in a list of supplementary resources that could be used to support teaching
• Rejected as unsuitable

Problems Experienced

Findings

The research team had difficulty in retrieving relevant material for the project using the search options in RDN subject gateways. Whenever two subject terms were combined the number of hits was drastically reduced. The search term "Heart" for example, retrieved 312 sites in BIOME; by adding a second term "Heart physiology" this was reduced to 8 sites. Search terminology was often restricted to key areas only, e.g. neoplasm, then the researchers trawled through the numerous hits to find materials – a lengthy process, but the only way to ensure that useful material had not been missed.

Searching under the sub-headings that had been provided by the academic staff produced few or in some cases no hits. A BIOME search for "ACE inhibitors" only retrieved 1 site. To provide enough material for the future strands of the project, Google was also used to locate materials for both mental illness and coronary heart disease/stroke.

On average only one in 10 of the resources located were passed to the academic staff for evaluation. The majority were too advanced, predominantly text based and therefore had no advantages over a textbook, or did not cover the project's subject areas (particularly the sub-headings/keywords).

Over 500 resources were evaluated by academic staff, but only 46% made it to the final repository of resources, or as a supplementary teaching resource. The main reasons for the rejection of the remainder were that the material was:

• Too advanced
• Not applicable/not helpful
• Repeating information found elsewhere
• US biased
• Not able to be accessed/opened.

Academic staff felt that, while some of the resources were excellent (particularly for cancer), in general the resources have not been as good as expected and there were not enough graphic or interactive materials for re-purposing. Mental health resources were geared towards the layperson and had a heavy emphasis on organisations.

Most of the resources went through a secondary evaluation stage to ensure that comments made by FE academic staff were applicable for HE and vice versa. In the secondary evaluation, there was general agreement between the academics in FE and HE about the usefulness of the resources. Although some materials were either too high or too low a level, others were rejected because of their similarity or due to problems of access.

All of the academics involved in the project, felt that they would use alternative sources to locate material. Google was their preferred option as it gave access to relevant material more easily than the subject gateways and has the advantage of advanced search strategies, including searches for images, applying multiple search terms, restricting searches by country of origin.

Conclusions and Recommendations

1. From a tutor's point of view, finding suitable resources via JISC repositories/subject gateways to support our target level of course may not be an attractive or viable option.
2. More resources are required to meet the needs of FE and the initial HE levels for the subjects covered by this project. In each of the key areas there were only 20 – 47 resources that could be used, with some element of re-purposing, to support teaching. The most common problem for exclusion from the repository was that the materials were either too advanced or not suitable for FE students.
   In the subject areas of mental illness coronary heart disease and stroke, the JISC resources alone were too few in number to support the other strands of work in the project and had to be supplemented by resources located using other search engines like Google.
   One of the objectives of our project is to look at ways of encouraging tutors to contribute their own resources to the digital repository which may assist to build a better bank of resources in the longer term. On a practical level, one of our specific deliverables is to prepare guidance on how to load resources into a digital repository.
3. The subject indexing and metadata tagging of resources needs to be more detailed to meet the needs of academic staff and learners. In areas where the staff knew they required images – e.g. the respiratory system – it was impossible to search for images alone. Searching involved going to each resource individually to check their contents.
   Many resource were rejected due to US bias and because they were not of a suitable quality to support a teaching situation. Improved metadata tagging could also assist with ensuring that tutors only access high quality material that will meet their selection criteria.
   We expect the search facilities provided by the JORUM repository will give us a lot more flexibility. Evidence from the repositories presented so far indicates that this will be the case.
4. The metadata tagging of resources in subject gateways would have to be amended to enable advanced search techniques to be applied. Some descriptions were more detailed than others, so some consistency is required.
   We believe that the metadata collected during this strand of our project is already of a much higher quality and will certainly address some of the problems described in the report.
5. The advanced search options in the subject gateways need to be improved as they very limited compared to standard search engines. As any searches for specific sub-headings or combinations of subjects produced very few results, the emphasis was on larger key areas only. With a hit rate of approximately 20 resources per day, this method would be a deterrent to academic staff with some experience of standard search engines.

Addressing points 3 and 4 above should significantly reduce the time tutors would have to spend on searching for resources.

References

1. Healthier Nation,
   <http://extranet.lauder.ac.uk/x4l>
2. Scottish Credit and Qualifications Framework,
   <http://www.scqf.org.uk/>

Contact Details

Heather Sanderson
Project Manager

Email: heather@kerson1.freeserve.co.uk