(As shown to the participants before the markets opened.)
Who will do the replications?
The replications will be conducted by research teams from Caltech, the University of Innsbruck, the Stockholm School of Economics and the National University of Singapore.
How were studies chosen for replication?
Please note that the selection of studies was done as mechanically as possible. Studies from AER and QJE were chosen because those are highly-selective journals; results reported there tend to be widely-cited and influential. Studies with student subjects were chosen to make replication easier in our labs. Only between-subject designs were chosen to have regularity in the nature of the designs.
How were hypotheses to be tested chosen?
From each study one between-subjects hypothesis is taken as the focus of replication, according to the following criteria:
Include the most “important” statistically significant between-subjects results/hypotheses in the experiment based on the results highlighted by the authors and our own reading.
If several statistically significant between-subjects results/hypotheses of about equal importance are in the paper we include the result/hypothesis concerned with efficiency.
If several statistically significant between subjects results/hypotheses still remain we randomly pick one.
Additional Background Information on the Replicated StudiesAdditional Information
Note that three of the studies (Kogan et al; Kuziemko et al; Marzilli Ericson and Fuster) included two separate experiments/studies which appear to have the same importance, as expressed by the authors' descriptions of their findings. These are clearly labeled as separate experiments rather than, for instance, as robustness tests of an initial central experiment. For these studies we included the second experiment and then applied criteria 1-3 above. The criteria of replicating the last experiment in papers which report a series of experiments has previously been used in the Open Science Framework Replication Project: Psychology.
For the following studies we randomly picked the hypothesis according to criterion 3:
De Clippel et al: it was randomly determined whether to replicate Pf2 or Pf4; and Pf2 was selected.
Duffy and Puzzello: it was randomly determined whether to replicate M6 vs NM6 or M14 vs NM14; and M6 vs NM6 was selected.
Dulleck et al: it was randomly determined whether to replicate B/LV vs B/V or B/L vs B/N; and B/LV vs B/V was selected.
Kogan et al: it was randomly determined whether to replicate insider groups vs control group or outsider groups vs control group; and insider groups vs control group was selected.
Kuziemko et al: it was randomly determined whether to replicate the 6 person or the 8 person redistribution experiment; and the 6 person redistribution game was selected.
What does it mean for a study to replicate?
Of course, prediction market design requires that event outcomes can be clearly said to have either occurred or not. Therefore, we define a result as being replicated if the statistical method used in the original paper yields a p-value <0.05. A result is not replicated if the corresponding p-value is >0.05.
How were sample sizes chosen?
All planned sample sizes were chosen so that if the replication produces an effect of the same strength as in the original study, the probability of p<0.05 in the replication is 90% (the tests have 90% power). In some cases, larger samples than those with 90% power were chosen because the lowest multiple of the typical original session size was somewhat above the 90% power sample.
How can we find out details of the replication procedure and test statistics?
Obviously, perfect replication is impossible. Therefore, our approach is to make every feasible effort to replicate all aspects of the design, software, and analysis as closely as possible (with the exception, in most cases, of different subject pools). Furthermore, we will be transparent about details of how the replications will take place. To this end, brief “replication reports” are available to market participants which describe the hypotheses and relevant differences between the original design and the planned replication. For more information on the replicated studies and the hypotheses see the “Included studies” page.