Last week we conducted a “Wizard of Oz study” for our Talking-Points design. This is "an experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being." [Wikipedia]

Method

We equipped two test participants with a cellphone with headset and let them walk down South University Street. We wrote a script with location information that we read to the participant over a second cellphone when he passed the respective locations. This information would later come from the social Talking-Points database. While the participant walked down the street we followed him with a video camera and observed how he would behave. We also recorded what the participant was saying as well as what the Talking-Points narrator was reading. After the experiment we gave the participants a short survey and interviewed them about their experience and suggestions. There were two different runs: on the first run we told the participant that he should use voice commands to control the systems on the second run he should use the keypad of his phone.

Data

So we ended up with two videos of the two runs we completed. Those videos we uploaded to Youtube for everyone to see.

In addition to that we have all the information we gathered from the surveys and the interviews.

Findings

We got very valuable insights on how to improve the design of our product as well as the design of this study.From the participants feedback it seemed that a voice controlled user interface would be a better than an interface through the cellphones keypad. We also got a lot of suggestions for new interactions with the system that we had not thought about in our initial design. For example, the user used the previously non existing command “repeat” because he couldn’t understand what the system was saying because of background noises. Another thing that showed to be really important would be a feature to skip ahead in the information the system is giving you.

A problem with the methodology of the study was that the narrator was walking to close behind the test person. The test person repeatedly tried to establish eye contact with the narrator when using voice commands. The task of the study was to get to TCF Bank on the Corner of South University and Church Street and see what kind of comments users had left about this location. We realized that in the future we would have to come up with a more realistic background task for the study. But all in all we were very pleased with the results of the WOz study and are planning to repeat it with blind or other sighted participants.