A usability test evaluation report on a web-based computer adaptive testing system

Kenneth Wong
Hong Kong Institute of Education

Reggie Kwan and Philip Tsang
Caritas Francis Hsu College

Yu Chi-wing
Po Leung Kuk Tang Yuk Tien College
Hong Kong SAR, China

This paper concentrates on the evaluation of a crucial feature of a web-based computer adaptive testing (CAT) system. The CAT system discussed, which was built on the concepts of the domain model and the Rasch model, aims to help teachers enhance the learning of K–S3 mathematics in Hong Kong. The two integrated models were used to provide adaptive features to students, such as navigation support, immediate feedback, an optimal study path and direct guidance. In order to discover any significant problems with the prototype, a usability test was performed at the development stage.

Several mathematics teachers and students were invited to evaluate the prototype, and their feedback on its user interface and pedagogical aspects was collected. Before conducting the usability test, the student participants were invited to complete a traditional paper-and-pencil test which contained 20 multiple-choice questions randomly selected from an item bank. Participants had 30 minutes to answer all the questions, and were not allowed to leave during this period. The test was carried out at least a week before the administration of the usability test to ensure that the students involved had not memorized the previous test questions.

During the usability test, all the participants were asked to:

• fill in a test user consent form and a questionnaire for demographic information;
• use a particular username and password to log on to the system online, having been given written instructions for accessing the CAT online;
• complete some tasks and finish the test; and
• answer a questionnaire.

This presentation indicates how best this test can be performed based on our experience and results.

During the usability test, a researcher videotaped and observed the performance of all the participants, who were encouraged to ‘think-aloud’ about what they were doing and why; and after the test, the system went through a fine-tuning or revamping process based on the test findings and was put to the usability test again. The usability test involved an iterative design, with the participants being asked to do it at least twice. The results showed that the CAT prototype provides a valuable, easy-to-use and fairly intuitive assessment tool, which can help to identify and diagnose students’ academic strengths and weaknesses.