John A List, Zacharias Maniadis, Fabio Tufano
Cited by*: Downloads*:

The sciences are in an era o fan alleged "credibility crisis'. In this study, we discuss the reproducibility of empirical results, focusing on economics research. By combining theory and empirical evidence, we discuss the import of replication studies, and whether they improve our confidence in novel findings. The theory sheds light on the importance of replications, even when replications are subject to bias. We then present a pilot meta-study of replication in experimental economics, a subfield serving as a positive benchmark for investigating the credibility of economics. Our meta-study highlights certain difficulties when applying meta-research (Ioannidis et al., 2015) and systematizing the economics literature.
John A List, Jeffrey A Livingston, Susanne Neckermann
Cited by*: Downloads*:

In the face of worryingly low performance on standardized test, offering students financial incentives linked to academic performance has been proposed as a potentially cost-effective way to support improvement. However, a large literature across disciplines finds that extrinsic incentives, once removed, may crowd out intrinsic motivation on subsequent, similar tasks. We conduct a field experiment where students, parents, and tutors are offered incentives designed to encourage student preparation for a high-stakes state test. The incentives reward performance on a separate low-stakes assessment designed to measure the same skills as the high-stakes test. Performance on the high-stakes test, however, is not incentivized. We find substantial treatment effects on the incented tests but no effect on the non-incented test; if anything, the incentives result in worse performance on the non-incented test. We also find evidence supporting the conclusion that the incentives crowd out intrinsic motivation to perform well on the non-incented test, but this effect is only temporary. One year later, students who had been in the incentives treatments perform better than those in the control on the same non-incented test.
Omar Al-Ubaydli, John A List
Cited by*: Downloads*:

This is a review of the literature of field experimental studies of markets. The main results covered by the review are as follows: (1) Generally speaking, markets organize the efficient exchange of commodities; (2) There are some behavioral anomalies that impede efficient exchange; (3) Many behavioral anomalies disappear when traders are experienced.
Omar Al-Ubaydli, John A List, Dana L Suskind
Cited by*: Downloads*:

Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether-and to what extent-insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people's lives, and a diminution in the public's trust in the scientific method's ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.
Greer K Gosnell, John A List, Robert D Metcalfe
Cited by*: Downloads*:

Increasing evidence indicates the importance of management in determining firms' productivity. Yet, causal evidence regarding the effectiveness of management practices is scarce, especially for high-skilled workers in the developed world. In an eight-month field experiment measuring the productivity of captains in the commercial aviation sector, we test four distinct management practices: (i) performance monitoring; (ii) performance feedback; (iii) target setting; and (iv) prosocial incentives. We find that these management practices -particularly performance monitoring and target setting- significantly increase captains' productivity with respect to the targeted fuel-saving dimensions. We identify positive spillovers of the tested management practices on job satisfaction and carbon dioxide emissions, and captains overwhelmingly express desire for deeper managerial engagement. Both the implementation and the results of the study reveal an uncharted opportunity for management researchers to delve into the black box of firms and rigorously examine the determinants of productivity amongst skilled labor.
Eszter Czibor, David Jimenez-Gomez, John A List
Cited by*: Downloads*:

What was once broadly viewed as an impossibility - learning from experimental data in economics - has now become commonplace. Governmental bodies, think tanks, and corporations around the world employ teams of experimental researchers to answer their most pressing questions. For their part, in the past two decades academics have begun to more actively partner with organizations to generate data via field experimentation. While this revolution in evidence-based approaches has served to deepen the economic science, recently a credibility crisis has caused even the most ardent experimental proponents to pause. This study takes a step back from the burgeoning experimental literature and introduces 12 actions that might help to alleviate this credibility crisis and raise experimental economics to an even higher level. In this way, we view our "12 action wish list" as discussion points to enrich the field.
Daniel J Benjamin, James O Berger, Magnus Johannesson, Brian A Nosek, E. J Wagenmakers, Richard Berk, Kenneth A Bollen, Bjorn Brembs, Lawrence Brown, Colin F Camerer, David Cesarini, Christopher D. Chambers, Merlise Clyde, Thomas D Cook, Paul De Boeck, Zoltan Dienes, Anna Dreber, Kenny Easwaran, Charles Efferson, Ernst Fehr, Fiona Fidler, Andy P. Field, Malcom Forster, Edward I. George, Tarun Ramadorai, Richard Gonzalez, Steven Goodman, Edwin Green, Donald P Green, Anthony Greenwald, Jarrod D. Hadfield, Larry V. Hedges, Leonhard Held, Teck Hau Ho, Herbert Hoijtink, James Holland Jones, Daniel J Hruschka, Kosuke Imai, Guido Imbens, John P.A. Ioannidis, Minjeong Jeon, Michael Kirchler, David Laibson , John A List, Roderick Little, Arthur Lupia, Edouard Machery, Scott E. Maxwell, Michael McCarthy, Don Moore, Stephen L. Morgan, Marcus Munafo, Shinichi Nakagawa, Brendan Nyhan, Timothy H Parker, Luis Pericchi, Marco Perugini, Jeff Rouder, Judith Rousseau, Victoria Savalei, Felix D. Schonbrodt, Thomas Sellke, Betsy Sinclair, Dustin Tingley, Trisha Van Zandt, Simine Vazire, Duncan J. Watts, Christopher Winship, Robert L. Wolpert, Yu Xie, Cristobal Young, Jonathan Zinman, Valen E. Johnson
Cited by*: 1 Downloads*: 965

We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.
Abhijit Banerjee, Angus Deaton, Esther Duflo
Cited by*: 28 Downloads*: 482

What are the determinants of the health and of well-being? Income and wealth are clearly part of the story, but does access to health-care have a large independent effect, as the advocates of more investment in health-care, such as the World Health Organization's Commission on Macroeconomics and Health (Commission on Macroeconomics and Health (2001)), have argued? This paper reports on a recent survey in a poor rural area of the state of Rajasthan in India intended to shed some light on this issue, where there was an attempt to use a set of interlocking surveys to collect data on health and economic status, as well as the public and private provision of health care.
John A List, Azeem M Shaikh, Yang Xu
Cited by*: 33 Downloads*: 278

Empiricism in the sciences allows us to test theories, formulate optimal policies, and learn how the world works. In this manner, it is critical that our empirical work provides accurate conclusions about underlying data patterns. False positives represent an especially important problem, as vast public and private resources can be misguided if we base decisions on false discovery. This study explores one especially pernicious influence on false positives-multiple hypothesis testing (MHT). While MHT potentially affects all types of empirical work, we consider three common scenarios where MHT influences inference within experimental economics: jointly identifying treatment effects for a set of outcomes, estimating heterogenous treatment effects through subgroup analysis, and conducting hypothesis testing for multiple treatment conditions. Building upon the work of Romano and Wolf (2010), we present a correction procedure that incorporates the three scenarios, and illustrate the improvement in power by comparing our results with those obtained by the classic studies due to Bonferroni (1935) and Holm (1979). Importantly, under weak assumptions, our testing procedure asymptotically controls the familywise error rate - the probability of one false rejection - and is asymptotically balanced. We showcase our approach by revisiting the data reported in Karlan and List (2007), to deepen our understanding of why people give to charitable causes.
John A List, Anya Samek, Dana L Suskind
Cited by*: 0 Downloads*: 258

Behavioral economics and field experiments within the social sciences have advanced well beyond academic curiosum. Governments around the globe as well as the most powerful firms in modern economies employ staffs of behavioralists and experimentalists to advance and test best practices. In this study, we combine behavioral economics with field experiments to reimagine a new model of early childhood education. Our approach has three distinct features. First, by focusing public policy dollars on prevention rather than remediation, we call for much earlier educational programs than currently conceived. Second, our approach has parents at the center of the education production function rather than at its periphery. Third, we advocate attacking the macro education problem using a public health methodology, rather than focusing on piecemeal advances.