Who will do the replications?

The replications will be conducted by research teams from Caltech, the University of Innsbruck, the Stockholm School of Economics and the National University of Singapore.

How were studies chosen for replication?

Please note that the selection of studies was done as mechanically as possible. Studies from AER and QJE were chosen because those are highly-selective journals; results reported there tend to be widely-cited and influential. Studies with student subjects were chosen to make replication easier in our labs. Only between-subject designs were chosen to have regularity in the nature of the designs.

How were hypotheses to be tested chosen?

From each study one between-subjects hypothesis is taken as the focus of replication, according to the following criteria:

  1. Include the most “important” statistically significant between-subjects results/hypotheses in the experiment based on the results highlighted by the authors and our own reading.
  2. If several statistically significant between-subjects results/hypotheses of about equal importance are in the paper we include the result/hypothesis concerned with efficiency.
  3. If several statistically significant between subjects results/hypotheses still remain we randomly pick one.

What does it mean for a study to replicate?

Of course, prediction market design requires that event outcomes can be clearly said to have either occurred or not. Therefore, we define a result as being replicated if the statistical method used in the original paper yields a p-value <0.05. A result is not replicated if the corresponding p-value is >0.05.

How were sample sizes chosen?

All planned sample sizes were chosen so that if the replication produces an effect of the same strength as in the original study, the probability of p<0.05 in the replication is 90% (the tests have 90% power). In some cases, larger samples than those with 90% power were chosen because the lowest multiple of the typical original session size was somewhat above the 90% power sample.

How can we find out details of the replication procedure and test statistics?

Obviously, perfect replication is impossible. Therefore, our approach is to make every feasible effort to replicate all aspects of the design, software, and analysis as closely as possible (with the exception, in most cases, of different subject pools). Furthermore, we will be transparent about details of how the replications will take place. To this end, brief “replication reports” are available to market participants which describe the hypotheses and relevant differences between the original design and the planned replication. For more information on the replicated studies and the hypotheses see the “Included studies” page.