Evaluation of the usability of electronic library resources (January 2000 - June 2001)

• Chief Investigators: Dr Kirsty Williamson and Dr Frada Burstein, School of Information Management and Systems, Monash University

• Funder: Australian Research Council SPIRT Program ($85,228 including $15,165 from the industry partner)

• Industry Partner: VICNET/State Library of Victoria

• Australian Post-graduate Award (Industry) – APA(I) student: Ms Nettie Handley
The major question, which this project set out to address is: What are the features which contribute to an attractive and usable electronic information resource?

Aims Included

• To evaluate four sets of electronic resources in relation to various factors including: ease of use; user preferences, and relevance of information content to users; and

• To develop comprehensive approaches for evaluation which include qualitative research methods, as well as some quantitative procedures developed from the data collected. The latter procedures will be implemented as a computerised tool.

The project set out to develop a comprehensive procedure for evaluation of the electronic resources which have been made available to users in the Victorian public library service through the Gulliver Program.

Methods

The collection of qualitative data involved four Victorian Public Library Services: Bayside, Port Phillip, Corangamite and Casey-Cardinia. Two of the library services are in metropolitan areas, one is a regional library service and one is in a semi-rural area. Each library service was allocated two of the four electronic resources (World Magazine Bank, Health Reference Library, Custom Database and Electric Library) for the purposes of the research. The research in each case began with a focus group of library staff - to explore their perceptions of content and usability of each of their two databases, as well as general access issues. A representative of SLV was included in each of the focus groups and attempts were made to include library staff in varying roles, e.g., a library manager and a library technician. The setting of homework, requiring staff to follow a topic of their choice in their two databases and to answer questions related to content and usability, resulted in data of high quality from these focus groups.

The second part of the fieldwork involved trials of the databases with 10 individual users in each of the library services. Requests were made to library users who were in the library at the time set aside for the trials, with attempts being made to involve the widest possible range of users in terms of age and gender.

A major finding of this project was that the major reason why the Gulliver databases were not being used to optimal level was not so much the quality of the databases offered – all were deemed to be of a high standard. Rather lack of training was blamed. This issue became the focus of a later ITNR project, ‘Developing the Competency of Australian Public Librarians in Using Online Databases’.

The qualitative data were supplemented with quantitative measurements, the latter being produced by a generic, multi-criteria evaluation tool. This tool was developed by researchers at Monash University and at The University of Melbourne to evaluate the success of decision support systems. The tool has a pre-defined set of evaluation criteria, e.g. ease of use, user satisfaction. These criteria were updated and adjusted to the new research context by key users of the resources (staff of public libraries and SLV/VICNET, as well as library patrons) through the qualitative data which was collected. This means that the resources were rigorously assessed from a 'multiple constituencies' point of view.

Current Status The qualitative data collection was completed in mid 2000. The data analysis was completed about three months later. Papers on this project were presented at the Reference and Information Services Section (RAISS) Symposium in October, 2001, (Melbourne) and at the Australian Library and Information Association Public Libraries Section Conference in November, 2001 (Melbourne). A summary report has been produced.

The component of the project which focuses on the computerised tool was completed and presented as a Masters thesis, written by the APA(I) student.