Effects of computer delivery mode on testing second language speaking : The case of monologic tasks
In recent years, computer-delivered assessment has gained prominence in large-scale tests of second language speaking. Despite its various advantages, the absence of an interlocutor in the computer-delivered speaking test has raised concerns over the effects of computer delivery mode on testing speaking. The present study thus addressed this issue by comparing examinees' performances (test score and speech sample) and their attitudes toward testing speaking, with a focus on monologic tasks delivered in the computer mode and the face-to-face mode.
The data used in this study was collected from 96 EFL students from two high schools and three universities in Japan. The examinees took two monologic tasks delivered in the computer mode and another two tasks with the same contents in the face-to-face mode in a counterbalanced design. After taking the tests, they completed two questionnaires, which included both Likert-scale items and open-ended questions on their attitudes toward the two modes. Examinees' responses to tasks were scored based on analytic scales of grammar, vocabulary, fluency, and pronunciation. Speech samples of the examinees were transcribed and coded for measures of fluency, accuracy, and complexity. In addition, the examinees were categorized into high, middle, and low-proficiency groups based on their scores in the computer-delivered monologic tasks.
Regarding examinees' test performance, consistent results were not obtained from the analyses of test score and speech sample. T-test results showed that there were no significant differences between the mean scores on any rating element or the total scores assigned to the computer and the face-to-face mode. The factor analysis did not reveal different psychometric constructs measured by the two modes either. These findings indicate that monologic tasks delivered in the two test modes could be considered to be equivalent in terms of test scores. However, the analysis of speech samples by means of two-way ANOVAs revealed significant but mixed effects of test mode on two measures of fluency. That is, examinees spoke more fluently with the computer mode in terms of repetition, while they were more fluent with the face-to-face mode in terms of filled pauses. Moreover, examinees of different proficiency levels were not found to perform differently across test modes.
Regarding the questionnaire results, it was found that examinees showed generally positive attitudes toward both the computer-delivered speaking test and the face-to-face test. Further, analyses by t-test and chi-square test revealed that examinees preferred the face-to-face mode, which was perceived to be more pleasant and more accurate in reflecting their spoken English, while the computer mode was considered as less anxiety-provoking. Based on the current findings, implications for language assessment, test development, and second language acquisition research were discussed, and directions for future studies were offered.