The replications will be conducted by research teams from Caltech, the University of Innsbruck, the Stockholm School of Economics and the National University of Singapore.
Please note that the selection of studies was done as mechanically as possible. Studies from AER and QJE were chosen because those are highly-selective journals; results reported there tend to be widely-cited and influential. Studies with student subjects were chosen to make replication easier in our labs. Only between-subject designs were chosen to have regularity in the nature of the designs.
From each study one between-subjects hypothesis is taken as the focus of replication, according to the following criteria:
Of course, prediction market design requires that event outcomes can be clearly said to have either occurred or not. Therefore, we define a result as being replicated if the statistical method used in the original paper yields a p-value <0.05. A result is not replicated if the corresponding p-value is >0.05.
All planned sample sizes were chosen so that if the replication produces an effect of the same strength as in the original study, the probability of p<0.05 in the replication is 90% (the tests have 90% power). In some cases, larger samples than those with 90% power were chosen because the lowest multiple of the typical original session size was somewhat above the 90% power sample.
Obviously, perfect replication is impossible. Therefore, our approach is to make every feasible effort to replicate all aspects of the design, software, and analysis as closely as possible (with the exception, in most cases, of different subject pools). Furthermore, we will be transparent about details of how the replications will take place. To this end, brief “replication reports” are available to market participants which describe the hypotheses and relevant differences between the original design and the planned replication. For more information on the replicated studies and the hypotheses see the “Included studies” page.