Experimental Economics Replication Project

Abeler et al. (AER 2011)

Abeler, Johannes, Armin Falk, Lorenz Goette, and David Huffman. 2011. “Reference Points and Effort Provision.” American Economic Review, 101(2): 470-92

Ambrus and Greiner (AER 2012)

Ambrus, Attila, and Ben Greiner. 2012. “Imperfect Public Monitoring with Costly Punishment: An Experimental Study.” American Economic Review, 102(7): 3317-32

Bartling et al. (AER 2012)

Bartling, Björn, Ernst Fehr, and Klaus M. Schmidt. 2012. “Screening, Competition, and Job Design: Economic Origins of Good Jobs.” American Economic Review, 102(2): 834-64

Charness and Dufwenberg (AER 2011)

Charness, Gary, and Martin Dufwenberg. 2011. “Participation.” American Economic Review, 101(4): 1211-37

Chen and Chen (AER 2011)

Chen, Roy, and Yan Chen. 2011. “The Potential of Social Identity for Equilibrium Selection.” American Economic Review, 101(6): 2562-89

de Clippel et al. (AER 2014)

de Clippel, Geoffroy, Kfir Eliaz, and Brian Knight. 2014. “On the Selection of Arbitrators.” American Economic Review, 104(11): 3434-58

Duffy and Puzzello (AER 2014)

Duffy, John, and Daniela Puzzello. 2014. “Gift Exchange versus Monetary Exchange: Theory and Evidence.” American Economic Review, 104(6): 1735-76

Dulleck et al. (AER 2011)

Dulleck, Uwe, Rudolf Kerschbamer, and Matthias Sutter. 2011. “The Economics of Credence Goods: An Experiment on the Role of Liability, Verifiability, Reputation, and Competition.” American Economic Review, 101(2): 526-55

Ericson and Fuster (QJE 2011)

Marzilli Ericson, Keith M., and Andreas Fuster. 2011. “Expectations as Endowments: Evidence on Reference-Dependent Preferences from Exchange and Valuation Experiments.” The Quarterly Journal of Economics, 126(4): 1879-1907

Author comment:

We applaud the effort of Camerer et al. (2016) to replicate studies in experimental economics. We were pleased to see that the results from the replication study strongly support the findings from Ericson and Fuster (2011). However, we believe the classification of our study as “not replicated” is not correct. Camerer et al. focus on the wrong statistical test in their replication classification, i.e. on a test that is both less powerful and more restrictive than the main specifications in our paper. Using the correct test, the replication results closely match the published results in magnitude and statistical significance.¹ Moreover, even using the incorrect test, the p-value is 0.055, quite close to the authors' cutoff for a successful replication.

Camerer et al. (2016)'s report of Ericson and Fuster shows that the effect studied is significant at p<0.01 in the paper’s preferred regression specifications (see Table 2, columns (2) to (5)). Moreover, the estimated effect size is almost identical to the results published in our paper. (Similarly, the simple comparisons of willingness to accept (WTA) for a mug, and also of log(WTA) across treatments, show differences that are significant at p<0.01.)

However, for classification purposes, Camerer et al. focus on a test that is less powerful than the regression and imposes undesirable functional form restrictions: the t-test for whether the difference in log(WTA) for a mug versus log(WTA) for a pen is the same in the low versus high expectations conditions. This t-test gives a p-value of 0.055, and hence counts as “not replicated” relative to the p=0.05 threshold. This is unfortunate, since we made it clear, in both Ericson and Fuster (2011) and our correspondence with the authors prior to the replication study when they asked for feedback on the replication design, that the regression-based results should be preferred not only on power grounds but also due to the more flexible functional form.

While the authors have noted that they needed to pick a single test to focus on for the purposes of their prediction market, we believe (both now and in advance) they picked the theoretically incorrect, less powerful test, and thus led our study to be misleadingly classified as “not having replicated”.

We are grateful for this project and its contributions to scientific knowledge, and want to clarify the record.

Sincerely,
Keith Marzilli Ericson and Andreas Fuster

¹ In our paper, the findings are summarized as “Result 2” (p. 1895): “Subjects who have a high chance of leaving with the mug value the mug significantly higher than subjects who have a low chance of leaving with the mug. We estimate that being in treatment MH instead of ML increases willingness-to-accept for the mug by 20–30%.”

Fehr et al. (AER 2013)

Fehr, Ernst, Holger Herz, and Tom Wilkening. 2013. “The Lure of Authority: Motivation and Incentive Effects of Power.” American Economic Review, 103(4): 1325-59

Friedman and Oprea (AER 2012)

Friedman, Daniel, and Ryan Oprea. 2012. “A Continuous Dilemma.” American Economic Review, 102(1): 337-63

Fudenberg et al. (AER 2012)

Fudenberg, Drew, David G. Rand, and Anna Dreber. 2012. “Slow to Anger and Fast to Forgive: Cooperation in an Uncertain World.” American Economic Review, 102(2): 720-49

Huck et al. (AER 2011)

Huck, Steffen, Andrew J. Seltzer, and Brian Wallace. 2011. “Deferred Compensation in Multiperiod Labor Contracts: An Experimental Test of Lazear's Model.” American Economic Review, 101(2): 819-43

Ifcher and Zarghamee (AER 2011)

Ifcher, John, and Homa Zarghamee. 2011. “Happiness and Time Preference: The Effect of Positive Affect in a Random-Assignment Experiment.” American Economic Review, 101(7): 3109-29

Kessler and Roth (AER 2012)

Kessler, Judd B., and Alvin E. Roth. 2012. “Organ Allocation Policy and the Decision to Donate.” American Economic Review, 102(5): 2018-47

Kirchler et al (AER 2012)

Kirchler, Michael, Jürgen Huber, and Thomas Stöckl. 2012. “Thar She Bursts: Reducing Confusion Reduces Bubbles.” American Economic Review, 102(2): 865-83

Kogan et al. (AER 2011)

Kogan, Shimon, Anthony M. Kwasnica, and Roberto A. Weber. 2011. “Coordination in the Presence of Asset Markets.” American Economic Review, 101(2): 927-47

Kuziemko et al. (QJE 2014)

Kuziemko, Ilyana, Ryan W. Buell, Taly Reich and Michael I. Norton. 2014. “Last-Place Aversion: Evidence and Redistributive Implications.” The Quarterly Journal of Economics, 129(1): 105-149

Replication reports

Abeler et al. (AER 2011)

Author comment:

Ambrus and Greiner (AER 2012)

Author comment:

Bartling et al. (AER 2012)

Author comment:

Charness and Dufwenberg (AER 2011)

Author comment:

Chen and Chen (AER 2011)

Author comment:

de Clippel et al. (AER 2014)

Author comment:

Duffy and Puzzello (AER 2014)

Author comment:

Dulleck et al. (AER 2011)

Author comment:

Ericson and Fuster (QJE 2011)

Author comment:

Fehr et al. (AER 2013)

Author comment:

Friedman and Oprea (AER 2012)

Author comment:

Fudenberg et al. (AER 2012)

Author comment:

Huck et al. (AER 2011)

Author comment:

Ifcher and Zarghamee (AER 2011)

Author comment:

Kessler and Roth (AER 2012)

Author comment:

Kirchler et al (AER 2012)

Author comment:

Kogan et al. (AER 2011)

Author comment:

Kuziemko et al. (QJE 2014)

Author comment: