The wisdom of crowds for visual search.

Juni and Eckstein show that perceptual decisions about large image data sets (as from medical and geospatial imaging) that are made by a group are more likely to be correct if group members' confidences are averaged than if a simple majority vote is taken:

Significance

Simple majority voting is a widespread, effective mechanism to exploit the wisdom of crowds. We explored scenarios where, from decision to decision, a varying minority of group members often has increased information relative to the majority of the group. We show how this happens for visual search with large image data and how the resulting pooling benefits are greater than previously thought based on simpler perceptual tasks. Furthermore, we show how simple majority voting obtains inferior benefits for such scenarios relative to averaging people’s confidences. These findings could apply to life-critical medical and geospatial imaging decisions that require searching large data volumes and, more generally, to any decision-making task for which the minority of group members with high expertise varies across decisions.

Abstract

Decision-making accuracy typically increases through collective integration of people’s judgments into group decisions, a phenomenon known as the wisdom of crowds. For simple perceptual laboratory tasks, classic signal detection theory specifies the upper limit for collective integration benefits obtained by weighted averaging of people’s confidences, and simple majority voting can often approximate that limit. Life-critical perceptual decisions often involve searching large image data (e.g., medical, security, and aerial imagery), but the expected benefits and merits of using different pooling algorithms are unknown for such tasks. Here, we show that expected pooling benefits are significantly greater for visual search than for single-location perceptual tasks and the prediction given by classic signal detection theory. In addition, we show that simple majority voting obtains inferior accuracy benefits for visual search relative to averaging and weighted averaging of observers’ confidences. Analysis of gaze behavior across observers suggests that the greater collective integration benefits for visual search arise from an interaction between the foveated properties of the human visual system (high foveal acuity and low peripheral acuity) and observers’ nonexhaustive search patterns, and can be predicted by an extended signal detection theory framework with trial to trial sampling from a varying mixture of high and low target detectabilities across observers (SDT-MIX). These findings advance our theoretical understanding of how to predict and enhance the wisdom of crowds for real world search tasks and could apply more generally to any decision-making task for which the minority of group members with high expertise varies from decision to decision.