Multimodal Error Correction for Speech-to-Text in a Mobile Office Automated Vehicle: Results From a Remote Study

Future users of automated vehicles will demand the ability to perform diverse and extensive non-driving related tasks. However, prevailing restrictions in the car require new interaction concepts to enable productive office work. Intelligent voice-based interfaces may be a solution to facilitate productivity while at the same time keeping the “driver in the loop” and thereby maintaining safety. In this work, we investigated the repair problem of productive speech-to-text input in a highly automated vehicle. We examined the user experience of selecting/navigating to an incorrectly recognized word using only speech, pointing and clicking on a touchpad, and using mid-air hand gestures. Results indicate that hand gestures (condition VaG) have high hedonic quality but are not considered viable for error correction in productive text input. On the other hand, the unimodal (Voice-only; baseline) and touchpad-based point-and-click (VaT) approaches to error correction were rated equally well in the hypothesized “mobile office” automated vehicle. The utilized remote study execution methodology proved to be a useful intermediary tool between pure online surveys and on-site studies for qualitative research during a pandemic but suffered from a lack of fidelity and options for objective usability and safety evaluation.