Metadata Harvesting

Background

As the number of available digital resources increases so does the need for quick and accurate resource discovery. In order to allow users to search more effectively many resource discovery services now operate across the resources of multiple distributed content providers. There are two possible ways to do this. Either by distributed searching across many metadata databases or by searching harvested metadata.

Metadata harvesting is the aggregation of metadata records from multiple providers into a single database. Building applications or services that use these aggregated records provides additional views of those resources, assisting in access across sectors and greater exposure of those resources to the wider community.

Open Archives Initiative Protocol for Metadata Harvesting

When metadata harvesting is carried out within the JISC Information Environment the Open Archives Initiative Protocol for Metadata Harvesting (OAI PMH) [1] version 2.0 is recommended. The Open Archives Initiative [2] had it roots in the e-prints community who were trying to improve access to scholarly resources. The OAI PMH was developed initially by an international technical committee in 1999. It is a light-weight low cost protocol that is built on HTTP and XML. The protocol defines six requests, known as verbs:

1. GetRecord Identify
2. ListIdentifiers
3. ListMetadataFormats
4. ListRecords
5. ListSets

In order for metadata to be shared effectively two things need to happen:

1. Content/data providers need to make metadata records available in a commonly understood form.
2. Service providers need to obtain these metadata records from the content providers and hold them in a repository.

OAI PMH provides a means of doing the above.

Record Format

At the lowest level a data provider must support the simple Dublin Core [3] record format ('oai_dc'). This format is defined by the OAI-PMH DC XML schema [4]. Data providers may also provide metadata records in other formats. Within the JISC Information Environment if the repository is of value to the learning and teaching community projects should also consider exposing metadata records that conform to the UK Common Metadata Framework [5] in line with the IMS Digital Repositories Specification using the IEEE LOM XML schemas [6] .

OAI-PMH also provides a number of facilities to supply metadata about metadata records for example rights and/or provenance information can be provided in the <about> element of the GetRecord response. Also collection-level descriptions can be provided in the <description> element of the Identify response.

Example OAI DC metadata record

The following example is taken from the Library of Congress Repository 1).

<oai_dc:dc>
<dc:title>Empire State Building. [View from], to Central Park</dc:title>
<dc:creator>Gottscho, Samuel H. 1875-1971, photographer.</dc:creator>
<dc:date>1932 Jan. 19</dc:date>
<dc:type>image</dc:type>
<dc:type>two-dimensional nonprojectible graphic</dc:type>
<dc:type>Cityscape photographs.</dc:type>
<dc:type>Acetate negatives.</dc:type>
<dc:identifier>http://hdl.loc.gov/loc.pnp/gsc.5a18067</dc:identifier>
<dc:coverage>United States--New York (State)--New York.</dc:coverage>
<dc:rights>No known restrictions on publication.</dc:rights>
</oai_dc:dc>

Conformance Testing for Basic Functionality

The OAI gives information on tests an OAI repository must successfully complete in order to be entered in the registry. For example:

• For every protocol request, the repository return a response that is valid XML (the XML successfully passes through an XML parser) and conforms to the XML schema defined for the response (passes the XSV XML schema validator).
• For the ListMetadataFormats request, the repository must return at least one metadata format and in the list of metadata formats, return the mandatory oai_dc metadata format with the URL of the OAI-defined XML schema.

More information on the tests necessary is available from the OAI Web site [7]. Projects could use the tests listed to create a checklist to measure their repository's conformance.

References

1. The Open Archives Initiative Protocol for Metadata Harvesting,
   <http://www.openarchives.org/OAI/openarchivesprotocol.html>
2. Open Archives Initiative,
   <http://www.openarchives.org/>
3. Dublin Core,
   <http://dublincore.org/>
4. OAI-PMH DC XML Schema,
   <http://www.openarchives.org/OAI/2.0/oai_dc.xsd>
5. UK Common Metadata Framework,
   <http://metadata.cetis.ac.uk/guides/>
6. IMS Digital Repositories Specification,
   <http://www.imsglobal.org/digitalrepositories/>
7. Registering as a Data Provider,
   <http://www.openarchives.org/data/registerasprovider.html>

Further Information

• OAI FAQ,
  <http://www.ukoln.ac.uk/distributed-systems/jisc-ie/arch/faq/oai/>
• OAI Software Tools,
  <http://www.openarchives.org/tools/>