Huaming Rao's Blog

(This post is originaly published in CrowdResearch.org, link .)

We created a spatial location identification task (SpLIT), in which humans examine a 3D camera view of an environment to infer its spatial location on a 2D schematic map (e.g. a floor plan). However, the SpLIT are often difficult to automate because the identification of salient cues often requires semantic features that are challenging to recover from the image and the map, especially when the cues are ambiguous or do not provide sufficient information to pinpoint the exact location.

One natural question to ask is how one can leverage human computation to perform a typical SpLIT. In order to find the answer, we designed a study using Amazon Mechanical Turk to investigate how turkers can perform the task in two kinds of reward schemes: ground truth and majority vote, even when they were not familiar with the environment.

We carefully chose five pictures represent different levels of ambiguity, in terms of the extent to which workers could use the pre-defined markers (cues) to infer the location of the camera view.

The results are listed below:

1. The accuracies of each task under both schemes drop as the level of ambiguity increases. This suggests that the lack of spatial cues did make the task harder. And it is obvious from the figure that performance in majority vote scheme was better than that in the ground truth reward scheme.

   

2. We also calculated the accuracies of individual participants, it showed that there were more turkers that at least chose one correct location in the majority vote scheme than in the ground truth one. In addition, the percentage with high ac- curacies (0.8 and 1.0) in the majority vote scheme is also higher than that in the ground truth one.

   

3. To summarize the results, we found that the majority vote reward scheme in general led to a higher level of precision and reliability in the answers provided by the workers.

   

Though there were not enough data to fully understand this result, the current study is clearly a first step towards understanding how human computation can be incorporated into applications that support spatial tasks

For more, see our full paper .

Author List:

• Huaming Rao , Huaming Rao, Nanjing University of Science & Technology and University of Illinois at Urbana-Champaign.
• Shih-Wen Huang , University of Illinois at Urbana-Champaign.
• Wai-Tat Fu , University of Illinois at Urbana-Champaign.