Ensembles Of Crowds And Computers: Experiments In Forecasting
Stevens Institute of Technology – Wesley J. Howe School of Technology Management
Stevens Institute of Technology – Stevens Institute of Technology – School of Business
Stevens Institute of Technology – School of Business
Stevens Institute of Technology – Stevens Institute of Technology – School of Business
June 25, 2015
Abstract:
This paper explores the power of news sentiment to predict financial returns, in particular the returns of a set of European stocks. Building on past decision support work going back to the Delphi method this paper describes a text analysis expert weighting algorithm that aggregates the responses of both humans and algorithms by dynamically selecting the best response according to previous performance. The proposed system is tested through an experiment in which ensembles of experts, crowds and machines analyzed Thomson Reuters news stories and predicted the returns of the relevant stocks mentioned right after the stories appeared. The expert weighting algorithm was better than or as good as the best algorithm or human in most cases. The capacity of the algorithm to dynamically select best answers from humans and machines results in an evolving collective intelligence: the final decision is an aggregation of the best automated individual answers, some of these come from machines, and some from humans. Additionally, this paper shows that the groups of humans, algorithms, and expert weighting algorithms have associated with them particular news topics that these groups are good at making predictions from.
Ensembles Of Crowds And Computers: Experiments In Forecasting – Introduction
Decision Support has at times relied on the opinions of people and at other times on computers. As online crowds and communities have shown potential in a variety of tasks, interest has increased in studying the reasons and applications of ensembles of humans, some expert, some not, in performing tasks and making forecasts. A related stream of research has looked at how crowd input can be used to increase machine learning. In the past, machine algorithms were under the control of humans. In some recent work, humans are under control of machines. That is, machines request humans to perform tasks; the output of those tasks are used as input to the machine algorithms. The new phenomena present challenges to current theories of information systems. For example, much of information systems has focused on the willingness of users to adopt a new technology. In some of the current work being done on crowds, the focus is on the extent to which computers should adopt the cognitive output of humans, whose performance is variable.
The current trends in research can lead to different conceptualizations of decision support, in which humans and computers are both viewed as participants in a complex decision process. Given that humans and computers have different cognitive skills and performance according to the Turing test (Turing 1950), how should tasks be allocated?
This general question can be addressed through a series of studies. We report the results of one such study here. We chose a cognitively complex task where ground truth exists. Specifically, we picked the domain of market forecasting, and within that, we asked both humans and computers to predict future stock prices based on past prices and past news stories.
Neither humans nor computers are able to perform this task with a high degree of accuracy (Creamer et al. 2013b): indeed, some argue that such markets are random walks (Fama 1965). However, many recent studies (Tetlock et al. 2008, Tetlock 2011) have shown that investors react at different speeds to the release of news information, so that it is at least, in theory, possible to make predictions about how investors will react to news in the near future. Several studies (Freund and Schapire 1997) have shown that machine algorithms can perform at slightly better than chance. This task is complex because it requires understanding news stories, and understanding how these stories may violate expectations of investors, leading to changes in investment strategies. Investing experience and domain knowledge of markets, companies, behavioral economics, and psychology may be helpful.
In sum, the task is one of anticipation that depends on a variety of skills. Success on the task can be readily measured. Thus, it provides a way of understanding the relative strengths of people and machines in a data-rich domain. This understanding, may, in turn, lead to new and different architectures for decision support systems related to forecasting markets. The findings may also provide a basis for further studies that might seek to determine how such architectures might work on different kinds of problems.
See full PDF below.
