On the Wisdom of Crowds:
Collective Predictive Analytics
“All great lies have a seed of truth” (James Cottrell, personal communication, 2004). In 1907, Sir Francis Galton (1855-1911) – a British statistician whose body of research focused on human intelligence and who also happened to be Charles Darwin’s cousin – observed that in a festival contest in Cornwall, where people attempted to guess the weight of an ox, the average of all guesses was consistently close to the ox’s actual weight (Galton, 1907; Ball, 2014; Gega, 2000). Author James Surowiecki resurrected this observation for his book, The Wisdom of Crowds, in 2005 (Surowiecki, 2005).
While Galton is often considered the “father” of modern collective intelligence based on a mean, his 1907 publication in the journal Nature focused on how well crowds predicted the median because it was subject to less error – or more confidence in its interval. The mean was within one pound; the median was within nine pounds, leading Galton to answer with more specificity to a reader’s letter that, because in small sets the exclusion of a measurement could greatly alter the mean and impact the median much less, he found collective intelligence most applicable for predicting median ranges rather than means (Galton, Letters to the editor: The ballot box, 1907). However, recent re-examinations of Galton’s data indicate that errors in the original calculations such that the 800-strong crowd’s estimate of the mean was not within one pound, but the exact weight of the ox (Wallis, 2014).
This theory of “collective intelligence” has been studied and analyzed for the 110 subsequent years to try and validate how, when, and in what circumstances it accurately and inaccurately predicts. Critics of Surowiecki’s popularizing review of collective intelligence point out crowds’ skill at optimizing while being less skilled at innovation or creativity (Lanier, 2010). Moreover, collective intelligence is poor at defining the right question – often the most important aspect of inquiry – and scalar results (Lanier, 2010).
One example of collective or aggregated intelligence’s predictive failure is in economic measurements. Experts who opine on measurements of economic growth are often incorrect and, moreover, this error is compounded because the prediction sets an expectation that, when missed, causes negative reactions in financial markets (Cassino, 2016). The failing of collective predictive intelligence in financial markets may largely be to this double-action – they set an expectation and the inaccuracy relative to the expectation makes markets respond disproportionately. While the origins and dynamics of irrational markets and “black swan events” are beyond the author’s scope here, these failings of collective intelligence may be an early causal event.
Other evidence suggests that collective intelligence can be effective creatively; however, mostly when applied to brainstorming-type activities among a cohort of experts. For example, in a 2011 contest in the Harvard Medical School community, 40,000 faculty, staff, and students competed to derive the most important questions – what is unknown but needed to be known – to cure type 1 diabetes, with impressive results (Harvard Medical School, 2011).
For now, the lessons learned regarding collective intelligence appear to be this: it is best when an intellectually diverse cohort of experts answer a predefined question and focuses on optimization around medians or brainstorming (Ball, 2014). It is least best when the crowd thinks the same, includes many non-experts, faces a spectrum or scalar of answers, and whose work product is not the basis for many future decisions on which error could be compounded.
Works Cited
Cassino, D. (2016, July 8). The ‘wisdom of the crowd’ has a pretty bad track record at predicting jobs reports. Havard Business Review, pp. https://hbr.org/2016/07/the-wisdom-of-the-crowd-has-a-pretty-bad-track-record-at-predicting-jobs-reports.
Galton, F. (1907). Letters to the editor: The ballot box. Nature, http://galton.org/cgi-bin/searchImages/galton/search/essays/pages/galton-1907-ballot-box_1.htm.
Galton, F. (1907). Vox populi. Nature, 450-451.
Gega, S. (2000, May). Sir Francis Galton. Retrieved from Muskingham College: http://muskingum.edu/~psych/psycweb/history/galton.htm
Harvard Medical School. (2011, April 6). The wisdom of crowds: Contest yields innovative strategies for conquering Type 1 diabetes. Retrieved from Harvard Medical School: https://hms.harvard.edu/news/wisdom-crowds-4-6-11
Lanier, J. (2010). You are Not a Gadget: A Manifesto. London: Allen Lane.
Surowiecki, J. (2005). The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations,. New York City: Anchor Books.
Wallis, K. (2014). Revisiting Francis Galton's forecasting competition. Statistical Science, 420-424.
