Papers - John A List
John A List
Cited by*: None Downloads*: None 
Last year I put together a summary of data from my field experiments website that pertained to artefactual field experiments. Several people have asked me if I have and update. In this document I update all figures and numbers to show the details for 2020. I also include the description from the 2019 paper.
John A List
Cited by*: None Downloads*: None
While empirical economics has made important strides over the past half century, there is a recent attack that threatens the foundations of the empirical approach in economics: external validity. Certain dogmatic arguments are not new, yet in some circles the generalizability question is beyond dispute, rendering empirical work as a passive enterprise based on frivolity. Such arguments serve to caution even the staunchest empirical advocates from even starting an empirical inquiry in a novel setting. In its simplest form, questions of external validity revolve around whether the results of the received study can be generalized to different people, situations, stimuli, and time periods. This study clarifies and places the external validity crisis into perspective by taking a unique glimpse into the grandest of trials: The External Validity Trial. A key outcome of the proceedings is an Author Onus Probandi, which outlines four key areas that every study should report to address external validity. Such an evaluative approach properly rewards empirical advances and justly recognizes inherent empirical limitations.
Daniel Hedblom,
Brent R Hickman,
John A List
Cited by*: None Downloads*: None
We develop theory and a tightly-linked field experiment to explore the supply side implications of corporate social responsibility (CSR). Our natural field experiment, in which we created our own firm and hired actual workers, generates a rich data set on worker behavior and responses to both pecuniary and CSR incentives. Making use of a novel identification framework, we use these data to estimate a structural principal-agent model. This approach permits us to compare and contrast treatment and selection effects of both CSR and financial incentives. Using data from more than 110 job seekers, we find strong evidence that when a firm advertises work as socially-oriented, it attracts employees who are more productive, produce higher quality work, and have more highly valued leisure time. In terms of enhancing the labor pool, for example, CSR increases the number of applicants by 25 percent, an impact comparable to the effect of a 36 percent increase in wages. We also find an economically important complementarity between CSR and wage offers, highlighting the import of using both to hire and motivate workers. Beyond lending insights into the supply side of CSR, our research design serves as a framework for causal inference on other forms of non-pecuniary incentives and amenities in the workplace, or any other domain more generally.
Robert T Ammerman,
Anne K Duggan,
John A List,
Lauren Supplee,
Dana L Suskind
Cited by*: None Downloads*: None
The goal of creating evidence-based programs is to scale them at sufficient breadth to support population-level improvements in critical outcomes. However, this promise is challenging to fulfill. One of the biggest issues for the field is the reduction in effect sizes seen when a program is taken to scale. This paper discusses an economic perspective that identifies the underlying incentives in the research process that lead to scale up problems and to deliver potential solutions to strengthen outcomes at scale. The principles of open science are well aligned with this goal. One prevention program that has begun to scale across the United States is early childhood home visiting. While there is a substantial impact research on home visiting, overall average effect size is .10 and a recent national randomized trial found attenuated effect sizes in programs implemented under real-world conditions. The paper concludes with a case study of the relevance of the economic model and open science in developing and scaling evidence-based home visiting. The case study considers how the traditional approach for testing interventions has influenced home visiting's evolution to date and how open science practices could have supported efforts to maintain impacts while scaling home visiting. It concludes by considering how open science can accelerate the refinement and scaling of home visiting interventions going forward, through accelerated translation of research into policy and practice.
John A List
Cited by*: None Downloads*: None
This review summarizes the historical place of the seminal contribution of Kahneman et al. (1990), from origins to theory to catalyst of an entire area of scholarship. This new literature has produced evidence both in concert with the original KKT conclusions as well as evidence refuting certain insights from KKT. The general theme of my summary is that even imperfect papers can have deep impact, both within and outside the academy; a lesson that today's critics should consider as young experimentalists continue to fight the tyranny of the top 5.
Marvin Cardoza,
Justin Holz,
John A List,
Alejandro Zentner,
Joaquin Zentner
Cited by*: None Downloads*: None
This paper uses a natural field experiment to examine the effectiveness of specific nudges on tax compliance amongst firms and the self-employed in the Dominican Republic. In collaboration with the Dominican Republic's tax authority, we designed messages for more than 28,000 self-employed workers and over 56,000 firms. Leveraging administrative tax data, we find evidence that our nudges (increasing the salience of prison sentences or public disclosure of tax evaders) have large effects on increasing tax compliance, primarily working through the channel of decreasing claimed tax exemptions. Interestingly, we find that firms are more impacted than the self-employed, and that firm size is critically linked to nudge effectiveness: larger firms are considerably more influenced by nudges than smaller firms. We find this latter result noteworthy given the paucity of evidence showing significant behavioral impacts of nudges amongst the largest players in a market. Overall, our messages increased tax revenue by $193 million (roughly 0.23% of the Dominican Republic's GDP in 2018), with over $100 million constituting income that the government would not have received without our field experimental nudges.
Luigi Butera,
Philip J Grossman,
Daniel Houser,
John A List,
Marie-Claire Villeval
Cited by*: None Downloads*: None
Creation of empirical knowledge in economics has taken a dramatic turn in the past few decades. One feature of the new research landscape is the nature and extent to which scholars generate data. Today, in nearly every field the experimental approach plays an increasingly crucial role in testing theories and informing organizational decisions. Whereas there is much to appreciate about this revolution, recently a credibility crisis has taken hold across the social sciences, arguing that an important component of Fischer (1935)'s tripod has not been fully embraced: replication. Indeed, while the importance of replications is not debatable scientifically, current incentives are not sufficient to encourage replications from the individual researcher's perspective. We propose a novel mechanism that promotes replications by leveraging mutually beneficial gains between scholars and editors. We develop a model capturing the trade-offs involved in seeking independent replications before submission of a paper to journals. We showcase our method via an investigation of the effects of Knightian uncertainty on cooperation rates in public goods games, a pervasive and yet largely unexplored feature in the literature.
Greg Allenby,
Russell Belk,
Catherine Eckel,
Robert Fisher,
Ernan Haruvy,
John A List,
Yu Ma,
Peter Popkowski Leszczyc,
Yu Wang,
Sherry Xin Li
Cited by*: None Downloads*: None
We offer a unified conceptual, behavioral, and econometric framework for optimal fundraising that deals with both synergies and discrepancies between approaches from economics, consumer behavior, and sociology. The purpose is to offer a framework that can bridge differences and open a dialogue between disciplines in order to facilitate optimal fundraising design. The literature is extensive, and our purpose is to offer a brief background and perspective on each of the approaches, provide an integrated framework leading to new insights, and discuss areas of future research.
Basil Halperin,
Benjamin Ho,
John A List,
Ian Muir
Cited by*: None Downloads*: None
We use a theory of apologies to analyze a nationwide field experiment involving 1.5 million Uber ridesharing consumers who experienced late rides. Several insights emerge. First, apologies are not a panacea: the efficacy of an apology and whether it may backfire depend on how the apology is made. Second, across treatments, money speaks louder than words - the best form of apology is to include a coupon for a future trip. Third, in some cases sending an apology is worse than sending nothing at all, particularly for repeated apologies. For firms, caveat venditor should be the rule when considering apologies.
Andreas Leibbrandt,
John A List
Cited by*: None Downloads*: None
Labor force composition and the allocation of talent remain of vital import to modern economies. For their part, governments and companies around the globe have implemented equal employment opportunity (EEO) regulations to influence labor market flows. Even though such regulations are pervasive, surprisingly little is known about their impacts. We use a natural field experiment conducted across 10 U.S. cities to investigate if EEO statements in job advertisements affect the first step in the employment process, application rates. Making use of data from nearly 2,500 job seekers, we find considerable policy effects, but in an unexpected direction: the presence of an EEO statement dampens rather than encourages racial minorities willingness to apply for jobs. Importantly, the effects are particularly pronounced for educated job seekers and in cities with white majority populations. Complementary survey evidence suggests the underlying mechanism at work is "tokenism", revealing that EEO statements backfire because racial minorities avoid environments in which they are perceived as regulatory, or symbolic, hires rather than being hired on their own merits. Beyond their practical and theoretical importance, our results highlight how field experiments can significantly improve policy making. In this case, if one goal of EEO regulations is to enhance the pool of minority applicants, then it is not working.
John A List,
Jeffrey A Livingston,
Susanne Neckermann
Cited by*: None Downloads*: None
In the face of worryingly low performance on standardized test, offering students financial incentives linked to academic performance has been proposed as a potentially cost-effective way to support improvement. However, a large literature across disciplines finds that extrinsic incentives, once removed, may crowd out intrinsic motivation on subsequent, similar tasks. We conduct a field experiment where students, parents, and tutors are offered incentives designed to encourage student preparation for a high-stakes state test. The incentives reward performance on a separate low-stakes assessment designed to measure the same skills as the high-stakes test. Performance on the high-stakes test, however, is not incentivized. We find substantial treatment effects on the incented tests but no effect on the non-incented test; if anything, the incentives result in worse performance on the non-incented test. We also find evidence supporting the conclusion that the incentives crowd out intrinsic motivation to perform well on the non-incented test, but this effect is only temporary. One year later, students who had been in the incentives treatments perform better than those in the control on the same non-incented test.
John A List
Cited by*: None Downloads*: None
No abstract available
Omar Al-Ubaydli,
John A List,
Dana L Suskind
Cited by*: None Downloads*: None
Policymakers are increasingly turning to insights gained from the experimental method as a means of informing public policies. Whether-and to what extent-insights from a research study scale to the level of the broader public is, in many situations, based on blind faith. This scale-up problem can lead to a vast waste of resources, a missed opportunity to improve people's lives, and a diminution in the public's trust in the scientific method's ability to contribute to policymaking. This study provides a theoretical lens to deepen our understanding of the science of how to use science. Through a simple model, we highlight three elements of the scale-up problem: (1) when does evidence become actionable (appropriate statistical inference); (2) properties of the population; and (3) properties of the situation. We argue that until these three areas are fully understood and recognized by researchers and policymakers, the threats to scalability will render any scaling exercise as particularly vulnerable. In this way, our work represents a challenge to empiricists to estimate the nature and extent of how important the various threats to scalability are in practice, and to implement those in their original research.
Eliot Abrams,
Jonathan Libgober,
John A List
Cited by*: None Downloads*: None
The past few decades have ushered in an experimental revolution in economics whereby scholars are now much more likely to generate their own data. While there are virtues associated with this movement, there are concomitant difficulties. Several scientific disciplines, including economics, have launched research registries in an effort to attenuate key inferential issues. This study assesses registries both empirically and theoretically, with a special focus on the AEA registry. We find that over 90% of randomized controlled trials (RCTs) in economics do not register, only 50% of the RCTs that register do so before the intervention begins, and the majority of these preregistrations are not detailed enough to significantly aid inference. Our empirical analysis further shows that using other scientific registries as aspirational examples is misguided, as their perceived success in tackling the main issues is largely a myth. In light of these facts, we advance a simple economic model to explore potential improvements. A key insight from the model is that removal of the (current) option to register completed RCTs could increase the fraction of trials that register. We also argue that linking IRB applications to registrations could further increase registry effectiveness.
Omar Al-Ubaydli,
John A List,
Claire Mackevicius,
Min Sok Lee,
Dana L Suskind
Cited by*: None Downloads*: None
Policymakers are increasingly turning to insights gained from the experimental method as a means to inform large scale public policies. Critics view this increased usage as premature, pointing to the fact that many experimentally-tested programs fail to deliver their promise at scale. Under this view, the experimental approach drives too much public policy. Yet, if policymakers could be more confident that the original research findings would be delivered at scale, even the staunchest critics would carve out a larger role for experiments to inform policy. Leveraging the economic framework of Al-Ubaydli et al. (2019), we put forward 12 simple proposals, spanning researchers, policymakers, funders, and stakeholders, which together tackle the most vexing scalability threats. The framework highlights that only after we deepen our understanding of the scale up problem will we be on solid ground to argue that scientific experiments should hold a more prominent place in the policymaker's quiver.
Alec Brandon,
John A List,
Robert D Metcalfe,
Michael K Price,
Florian Rundhammer
Cited by*: None Downloads*: None
This study considers the response of household electricity consumption to social nudges during peak load events. Our investigation considers two social nudges. The first targets conservation during peak load events, while the second promotes aggregate conservation. Using data from a natural field experiment with 42,100 households, we find that both social nudges reduce peak load electricity consumption by 2 to 4% when implemented in isolation and by nearly 7% when implemented in combination. These findings suggest an important role for social nudges in the regulation of electricity markets and a limited role for crowd out effects.
Glenn W Harrison,
John A List
Cited by*: 644 Downloads*: 105
Experimental economists are leaving the reservation. They are recruiting subjects in the field rather than in the classroom, using field goods rather than induced valuations, and using field context rather than abstract terminology in instructions. We argue that there is something methodologically fundamental behind this trend. Field experiments differ from laboratory experiments in many ways. Although it is tempting to view field experiments as simply less controlled variants of laboratory experiments, we argue that to do so would be to seriously mischaracterize them. What passes for "control" in laboratory experiments might in fact be precisely the opposite if it is artificial to the subject or context of the task. We propose six factors that can be used to determine the field context of an experiment: the nature of the subject pool, the nature of the information that the subjects bring to the task, the nature of the commodity, the nature of the task or trading rules applied, the nature of the stakes, and the environment that subjects operate in.
Steven D Levitt,
John A List
Cited by*: 321 Downloads*: 96
A critical question facing experimental economists is whether behavior inside the laboratory is a good indicator of behavior outside the laboratory. To address that question, we build a model in which the choices that individuals make depend not just on financial implications, but also on the nature and extent of scrutiny by others, the particular context in which a decision is embedded, and the manner in which participants and tasks are selected. We present empirical evidence demonstrating the importance of these various factors. To the extent that lab and naturally occurring environments systematically differ on any of these dimensions, the results obtained inside and outside the lab need not correspond. Focusing on experiments designed to measure social preferences, we discuss the extent to which the existing laboratory results generalize to naturally-occurring markets. We summarize cases where the lab may understate the importance of social preferences as well as instances in which the lab might exaggerate their importance. We conclude by emphasizing the importance of interpreting laboratory and field data through the lens of theory.
John A List
Cited by*: 262 Downloads*: 100
This study examines individual behavior in two well-functioning marketplaces to investigate whether market experience eliminates the endowment effect. Field evidence from both markets suggests that individual behavior converges to the neoclassical prediction as market experience increases. In an experimental test of whether these observations are due to treatment (market experience) or selection (e.g., static preferences), I find that market experience plays a significant role in eliminating the endowment effect. I also find that these results are robust to institutional change and extend beyond the two marketplaces studied. Overall, this study provides strong evidence that market experience eliminates an important market anomaly.
Uri Gneezy,
John A List
Cited by*: 262 Downloads*: 31
Recent discoveries in behavioral economics have led scholars to question the underpinnings of neoclassical economics. We use insights gained from one of the most influential lines of behavioral research -- gift exchange -- in an attempt to maximize worker effort in two quite distinct tasks: data entry for a university library and door-to-door fundraising for a research center. In support of the received literature, our field evidence suggests that worker effort in the first few hours on the job is considerably higher in the "gift" treatment than in the "non-gift treatment." After the initial few hours, however, no difference in outcomes is observed, and overall the gift treatment yielded inferior aggregate outcomes for the employer: with the same budget we would have logged more data for our library and raised more money for our research center by using the market-clearing wage rather than by trying to induce greater effort with a gift of higher wages.