Learning Across Alternatives

A Modeling Approach for Decision Making in Complex Environments


In many settings agents face difficult problems, be they managers, policymakers, judges, or consumers. A significant part of the difficulty is that decision makers face many options from which to choose yet possess only a tenuous understanding of how these alternatives map into outcomes.  In my work, I have introduced an approach to modeling problems of this sort and, along with co-authors, explored its application to a variety of settings in economics, political science, and management. Other researchers have picked up this approach, advancing our understanding further and pushing the approach into new areas.

The key novelty of the approach is to represent the mapping from actions to outcomes as the realized path of a Brownian motion. The mapping is fixed throughout time and the agents learn about the mapping through experience, by consulting an expert, by sampling actions, and so on. No matter how many (finite) action-outcome pairs they observe, however, there is always more to learn, and the agents never learn their environment completely. 

The Brownian motion representation, despite its apparent abstraction, has surprisingly attractive and realistic learning properties. The variance of the Brownian motion captures the complexity of the decision-making environment. The information space is rich—there is a continuum of state variables (one for each action)—but the states are correlated. The difficulty in learning about an environment depends on the degree of this correlation. Thus, the variance of the Brownian motion—how correlated are nearby alternatives–measures the complexity of the decision-making environment. In an important precursor to my work, Jovanovic and Rob (Econometrica, 1993) show that the Brownian formulation emerges from a set of natural axioms on the nature of uncertainty.

Learning in the Brownian motion framework can be best understood through the notion of informational spillover. Suppose an agent learns the outcome of one particular action. This can be thought of as practical knowledge – she knows what outcome a particular action will produce. In addition to this practical knowledge, the agent also possesses theoretical knowledge – she knows how the world works. She can then combine her practical with her theoretical knowledge to draw inferences about other actions. Her practical knowledge tells her that the mapping from actions to outcomes passes through a particular point.  Knowing that the mapping is generated by a Brownian motion pins down her beliefs about the other actions. In this way, information from knowing the outcome of one action spills over to other actions.

The informational spillover in the Brownian environment is positive but imperfect. This means that the environment is partially invertible. Knowledge of the Brownian path spills over to other actions, shaping a decision maker’s beliefs about the likely outcome.

The Brownian motion also possesses the appealing and intuitive property of proportional invertibility – an agent learns more about actions that are close to the known point and less about actions that are more distant. The level of informational spillover – the degree of invertibility – is parameterized by the complexity of the environment. The more complex is the environment – the higher the ratio of variance to drift of the Brownian motion – the lower is the informational spillover.

As we will see below, these properties contrast with those standard in the literature. The canonical assumption in models of experimentation is that the arms of the bandit are be independent, which implies there is no informational spillover whatsoever. At the other end of the spectrum lies models of strategic communication. There the canonical assumption is for informational spillover to be complete. Knowledge of one point in the mapping implies knowledge of all points in the mapping. In these models, expertise is perfectly invertible. These differences matter for how we understand the role of information in economic decision making.

My approach represents a non-standard use of the Brownian motion. In particular, the Brownian motion does not evolve through time. (One may think of the process as having evolved through the action space at time 0 before play begins). I leverage the mathematics of the Brownian motion to provide a concise and rich representation of uncertainty and learning in complicated environments.

I have used the Brownian motion representative to explore a variety of problems, touching upon applications otherwise considered distinct. Unifying these applications with a common model of uncertainty is an objective of my work. I emphasized this unified perspective in a short course I presented in 2020. In what follows I describe the questions and areas of economics to which I, and others, have applied the Brownian approach to decision making under uncertainty across various domains.

1. Learning from Experience

A starting point is the classic question of experimentation: When do agents seek safety in repeating past (and known) actions versus experimenting with a risky but potentially superior action? This is formalized as the classic bandit problem in which there is a set of discrete and independent (uncorrelated) arms. The Brownian motion approach allows for the arms to be correlated and for there to be a continuum of them that are deterministic. Thus, the question is not “how long” to pursue an arm, where the arms are independent and essentially indistinguishable. Rather, the question is which arm should I choose given what I have learned so far?

The Brownian motion approach is appealing also as it offers a notion of distance. How “far” is an experiment from what is known? Thus, the approach opens up new questions, such as How bold is the experiment when it does happen and in which direction do agents search? Does learning ultimately converge on good alternatives or can it get stuck at inefficient action? How is learning affected, be it distorted or improved, when agents have different preferences?

 

Experimentation, Search, & Learning:

In “Searching and Learning by Trial and Error,” I  explore optimal search and characterize the equilibrium path of learning over time. I show how search falls into two natural phases before ultimately stabilizing. A key finding – and difference with the classic model of search over independent alternatives – is that search can return to a previously discarded alternative and get “stuck,” leading to substantial inefficiency. 

 I then show that these patterns match the famed “product life cycle” in markets where the search is performed by firms seeking out an optimal product design. In the first phase of search – the monotonic phase – firms expand the set of products that are explored,  searching in new parts of the product space and always in the same direction. Eventually the monotonic phase ends and the second phase – the triangulating phase – begins. In the triangulating phase, search iterates within the boundaries of existing products, repeatedly changing direction, as firms narrow in their focus on a part of the space that is particularly promising. The phases can each last for arbitrary numbers of periods, although both phases, and search itself, ultimately ends with probability one.

 An assumption in the AER paper is that the firms have a target outcome they are seeking to hit and they know what this target is. In many economic settings, more is better, and there is no specific target beyond that. In “The Risk of Failure,” we adapt the search model to the allow for these preferences. In this environment search is constrained only by risk aversion, and we examine the impact of risk on experimentation and what it means for long-run performance.

 

Forward-Looking Search

An appeal of the Brownian motion approach is that, in many ways, it is highly tractable. The richness of uncertainty does come with some trade-offs. One trade-off is that it is difficult is to capture forward looking behavior. In the two papers above I assume the agents are myopic. 

In “Preemptive Policy Experimentation,” we extend the agents’ planning horizon to two periods and show how the long shadow of policy  experimentation can significantly shape current policy choices.

In two other papers, I provide partial results on forward looking behavior.  In “The Risk of Failure,” listed above, we show how a two-period
planning horizon does not qualitatively affect search behavior. Similarly, in “Agenda Control under Policy Uncertainty,” with Nolan McCarty in the AJPS (listed below), we show how the longer horizon can have no effect on search and experimentation in a strategic environment, in contrast to the usual experimentation intuition that a longer horizon encourages more experimentation.

Much more progress on forward-looking search has been made by other researchers.

  • Bruno Strulovici and Umberto Garfagnini, “Social Experimentation with Interdependent and Expanding Technologies,” Review of Economic Studies, 2016.

Bruno and Umberto apply the Brownian structure to study overlapping generations of agents where each generation has a two-period planning horizon. They show how this can generate waves of innovation through time.

For agents with unbounded horizons, deep progress has been made when search is  continuous. In continuous search, an agent must search along the alternatives as they are ordered. An example of continuous search is a mining company searching over a patch of ground for oil, gold, or other minerals. The agent does get to control the speed, or intensity, of the continuous search. These papers are able to characterize optimal far-sighted search behavior. One particularly interesting finding in the work of Urgun and Yariv is that the optimal speed of search is U-shaped, being highest when an agent is approaching a breakthrough or on the verge of giving up. Wong connects these results to the size of firms and establishes the conditional Gibrat’s law in which the size of a firm and its profitability are related linearly.

 

Two-dimensional Search: The Brownian Staple

 ·       “Recombinant Search.” Working paper, with Arjada Bardhi, 2024.

In recent work with Arjada Bardhi, we have been exploring a novel two-dimensional generalization of the Brownian motion – what we refer to as the Brownian staple. This generalization allows us to study recombinant search, where agents can combine ideas from across multiple fields, domains, or markets, to come up with new insights.

 

Experimentation & Learning in Policymaking and Competitive Elections:

How do complex environments affect strategic behavior when agents have different preferences over the outcomes produced? I explore these questions in the domain of political economy through the classic lens of political parties competing over policy.

In “Searching for Good Policies” I show how competitive elections can facilitate more effective experimentation and produce better long-term outcomes for voters, even when those parties are partisan with different preferences to voters.

In “Preemptive Policy Experimentation” we show how the likelihood of being succeeded in office by a party with different preferences can induce policy experimentation where it otherwise wouldn’t occur. The early policymaker, in a sense, experiments so as to teach his successor about the policy mapping, hoping to shape his successor’s policy choice. This result shows how policy choices can cast long shadows over the future even when the linkage across time is purely informational.

In “Agenda Control Under Policy Uncertainty,” we apply the Brownian structure to Romer and Rosenthal’s classic agenda setter model. We show how Brownian policy uncertainty upends many of Romer and Rosenthal’s classic insights and expands the classic gridlock interval.

An interesting implication of agenda-setting power is that the linkage across periods is not just informational – as it is in all papers described above – as last period’s policy choice is privileged as this period’s status quo and, thus, default option. This captures the importance of institutional features in structuring optimal search behavior. Inter-period linkages that are more than informational are also central to my work on competition in markets that I describe below (Section 4).

Several political scientists & political economists have pursued the Brownian motion approach, representing different aspects of how the difficulty and complexity of policy issues affects the operation of political institutions.

See also Tom Clark’s paper in the Journal of Theoretical Politics in Section 3.

 

 An Application to Management of Organizations

In a (long-percolating) working paper with Niko Matouschek, “Managing on Rugged Landscapes,” we apply the Brownian model to learning within firms. We focus on the challenges of managerial search over a complicated space of strategies when managers must coordinate their actions inside the firm and can observe and imitate competitors.

2. Strategic Communication

The Brownian motion can be applied to questions of strategic communication. Information is asymmetric with the expert knowing the state of the world. In contrast to the canonical models in which expertise is a single piece of information (Crawford and Sobel 1982; Milgrom 1981), the expert knows the full path of the Brownian motion. In this line of work I study these “complex environments” and how strategic communication between an expert and a decision maker can change relative to the simple expertise of the canonical models.  These models are one-shot rather than dynamic. 

Cheap Talk:

In these papers I study the canonical model of cheap talk of Crawford and Sobel (1982) when expertise is complex. We show how communication can be efficient — with no wasted information — and sender-optimal when expertise is complex. Both of these properties contrast with the partitional equilibria of simple environments. The key insight is that in complex environments experts are able to communicate precisely yet imperfectly. Because the expert’s informational advantage is large, she can recommend her most preferred action without giving up all of her information to the receiver. The expert can use her information and at the same time keep some of it private. We identify a strategy by which the sender can do this and control the spillover of information so that she can retain leverage over the decision. Indeed, we show that the equilibrium outcome is the same as it would be if the expert held full decision making authority.

The working paper with Yunus Aybas, a PhD student at Stanford, provides a characterization of a sender-optimal equilibria for small and large bias, showing how it depends on the size of the action space and the nature of the informational spillover from the sender’s recommendation. By identifying the key ingredients to sustain efficient cheap talk, it extends beyond the Brownian motion environment and describes other complex environments in which efficient cheap talk is possible. 

The earlier paper, “A Theory of Policy Expertise,” was the first paper I wrote using the Brownian motion framework. It identifies the equilibrium for small bias only and only when the action space is unbounded. It then focuses on how the different nature of advice creates an incentive for the sender to acquire expertise, an incentive that isn’t present in the same way in the traditional Crawford-Sobel setting. Interestingly, the agent who has the most incentive to acquire expertise is an agent with positive bias (thus, different outcome preferences to the decision maker). This shows why strategic communication may often be between players with divergent preferences.

Hard (Verifiable) Information:

We show that in complex environments, the nature of communication qualitatively changes. With information verifiable, experts now provide advice that is more than just a recommendation. They provide “referential” as well as “conative” advice. That, unlike existing models, experts communicate additional information about the environment in addition to a recommendation. We show that by providing this additional information, and only by providing this additional information, is the expert able to leverage her expertise and sway decisions to her own advantage.

 

Other Equilibria. A limitation of my work in this area is that we characterize only some equilibria, focusing on those that deliver leverage to the expert. A more complete exploration of equilibria, particularly in the cheap talk setting, is a natural and needed direction for follow-on work. Identifying other equilibria is important to understanding the nature of strategic communication in complex environments. Additionally, it will help create a bridge between equilibria in the simple environments of the literature and complex environments that I have studied.

 

 

Learning from Experience & from an Expert: A Unified Approach

  • “Experts & Experiments,” with Yunus Aybas, working paper, 2024.

In economics, research into search & experimentation has proceeded in parallel but entirely separately from research into strategic communication and the role of experts in decision making. The Brownian motion approach — in being able to address questions of experimentation and expert advice — offers a way to unify the questions that underlie these research streams.

Consider the problem of a policymaker dissatisfied with the current level of the minimum wage. He sees the current wage of, say, $10 an hour, as being inadequate, as not providing a “livable” wage. This is an obvious and topical problem of decision-making under uncertainty. Increasing the minimum wage puts more money into the pockets of workers, but if the wage is increased too much, it may drive businesses away, reducing the number of workers, and lowering the aggregate welfare of workers. That the effects of the minimum wage has been a flourishing – and often heated – debate in academic economics emphasizes the uncertainty surrounding this question.

In this setting, the policymaker has (at least) two options for gathering information to make a better decision. He can ask an expert for advice, or he can try a wage level, see if it works, and learn from his experience. He can also do both. He can ask an expert for advice and experiment. A set of natural — and new — questions then emerge. How does expert advice shape the policymaker’s experiment? In turn, how does the prospect of experimentation shape the advice the expert provides? Do expert advice and experimentation coexist in equilibrium?

The duality of experts & experiments is not unique to policymaking. In almost any setting with either an expert or experimentation, both options are actually present. This includes most, if not all, examples used to motivate the literatures on expert advice and experimentation.

In an ongoing project with Yunus Aybas, we study this problem in a two-period model that builds on the canonical model of cheap talk. We introduce an environment that is inspired by the Brownian motion but captures the essential property of partial invertibility in a simpler, more tractable representation. 

3. Strategic Learning when Facing a Flow of Problems

Common to the experimentation and communication papers is that the decision maker seeks a single action that delivers the best outcome to maximize her utility. Any other learning is only incidental and a means-to-this-end. In many decision making environments, the players care about the outcome of many actions, possibly even the entire function. I explore this type of problem in the paper:

This paper considers the problem of courts facing a flow of related cases on which verdicts must be rendered. The cases are related but differ ever so slightly or significantly from each other. As much as we’d like to think of judges handing down justice impartially and without error, the reality is much different. Judges must issue judgements with incomplete information and imperfect precedents to guide their thinking.

We use the Brownian motion to capture this problem faced by the courts. Each point in the real line corresponds to a set of case facts that can appear in front of a judge. The judge must determine the correct judgement corresponding to each set of case facts, whether that be guilty or innocent, liable or not liable, etc. The problem is that the courts do not know the mapping of real-world facts into legal outcomes. We suppose a standard judicial hierarchy, and that the Higher Court can learn the legal outcome for any case, but that it has limited resources and can hear only a single case in each period. The flow of cases–a continuum, we assume–must be heard by the Lower Courts, who cannot discern correct legal outcomes, but do have available the precedents issued by the Higher Court. The questions we explore are: (1) How do the Lower Courts adjudicate in such an informationally complex environment? (2) How does this equate with the common, but much criticized, practice of analogical reasoning? (3) What does these answers imply for how the Higher Court chooses to hear cases and how it shapes its doctrine to guide Lower Court decision making?

The greatest progress in this area has been made by other researchers using the Brownian motion framework. 

In “Attributes,” Arjada Bardhi provides a thorough and impressive analysis of the fundamental problem. To consider one natural and important example: Consider a pharmaceutical company that seeks approval for a new drug from the FDA. Suppose the drug works differentially on different segments of the population, as all drugs do, and that the FDA’s standard of proof is different than that of the pharmaceutical company, as is also true. In conducting clinical trials to prove the efficacy of the drug, which sub-populations should the pharmaceutical company test the drug on? Arjada analyzes this problem in a very general setting, beginning with the Brownian motion but extending beyond it to a broad class of stochastic processes. In so doing, this analysis shows that the Brownian motion representation is an example (a tractable one with appealing stationary and linear properties) of a broader domain that can be used in, and fitted to, particular applications.

Arjada and Nina Bobkova use somet of these techniques to study the intriguing idea of mini-publics. This is an idea popular in political science, philosophy, and in practice, whereby a (hopefully) representative set of citizens are drawn to make policy decisions. The novel idea is that the citizens are sampled according to some process rather than elected by the public at large. This idea had not been subject to much formal analysis, and not a level even close to appropriate given the debates in society about the idea. Arjada and Nina applied the Brownian motion representation, and other stochastic processes, to study the problem rigorously, characterizing the optimal way to sample the citizen representatives.

In a rather different vein, the exciting paper by Johannes Schneider and Christoph Carnehl applies this framework to the question of  how scientists “stand on the shoulders of giants” to advance our knowledge. An intriguing idea within their model is that scientists have to choose questions to ask and a region in which to look for the answer. This implies that not all research questions lead to answers, and the absence of an answer leads to an inference problem for later researchers. Johannes and Christoph characterize the progress of science in this setting. One of their nicest results is to characterize the role of “moon shot” research projects, when they are optimally undertaken, and how they can jolt a scientific field out of a rut and back onto an advancing path. 

4. Market Competition

Innovation is integral to economic growth. A long standing question in the Industrial Organization (IO) literature is the extent to which market structure incentivizes innovation. By its very nature, innovation is speculative and uncertain, with many new technologies, products, and potential markets to try. In recent work, we have begun to apply the Brownian motion framework to this problem to understand how the difficulty in finding successful innovations shapes the strategy of firms and the outcomes of market competition. 

In “The Novelty of Innovation,” we develop a simple model of entry by a new firm. In using the Brownian motion to represent the quality of potential innovations, we are able to capture a continuous notion of distance between technologies that has not appeared in the literature. This allows us to get at the question of not whether innovation occurs but what innovations are pursued and succeed in the market. We explore the implications of the model for the application of antitrust policy in innovative industries.

In “Innovation and Competition on a Rugged Technological Landscape,” we explore this model dynamically. In each period a new firm decides whether to enter the market and, if so, where to locate. For the innovation decision of each firm, this provides a rich choice. In addition to a bold or an incremental innovation, the firm decides whether to innovate in a niche, within the boundaries of the extant market, or beyond the frontier, expanding the scope of the market. We characterize the innovation strategies of the individual firms and the dynamic path and efficiency of the market over time.

Our AEA Papers & Proceedings paper summarizes and expands upon the ideas in both of these papers in a concise form.

The key technical novelty of these papers is that the linkage across periods is no longer purely informational. The payoff a firm receives depends on the outcome of the mapping at their location and also the location of the other firm(s) that it must compete with. In this way, innovation and competition interact and shape the market strategies of the firms. The firms are no longer just searching, they are searching and competing simultaneously.

As mentioned above, the political economy paper “Agenda Control under Policy Uncertainty” contains a two-period extension that also has this feature. In that context, the linkage across time is that the policy choice today becomes the status quo tomorrow and, given the collective choice requirement, this shapes future policymaking possibilities. This connects to the political economy literature on an endogenous status quo.

Learning Across Alternatives — A Short Course

In October 2020, I delivered a short course on my work in this area. The title of the course was “Learning Across Alternatives: Strategic Behavior in a Complicated World.” The short course was delivered in conjunction with the Org Econ Workshop organized annually in Sydney (a great conference! You should consider attending in the future). The course was broken down along the lines of the four decision problems listed above. The slides for the lectures are available here:

Formal Structure of the Model

Formally, the Brownian motion representation corresponds to a bandit problem with a continuum of correlated, deterministic bandits. The key feature is the correlation. The classic assumption in the experimentation literature is of independent bandits, typically with stochastic outcomes (and usually with a single, or at most finite, risky arm). The question the literature explores is how long to persist with an arm before abandoning it. Notably, when an arm is abandoned, nothing new is learned about the other arms and search begins again from scratch. With correlation across the arms this property is no longer true. In the Brownian framework, every experiment – be it a success or a failure – informs future choice. A success guides the way to where good alternatives are to be found, and a failure the opposite. This interdependence of outcomes leads to rich and path dependent behavior. Yet it does come (for the moment at least) with an analytic cost. In most of my search papers I assume agents are short-sighted, optimizing utility on a per-period basis. The exceptions are “Preemptive Policy Experimentation” that extends the horizon to two periods, and “The Risk of Failure,” that shows the structure of myopic search is robust to far-sighted agents. It is well known from the bandit literature that correlation is difficult to handle analytically. Nevertheless, the Brownian framework provides enough structure on beliefs (particularly through the Markov property) that the model with far-sighted behavior may yet prove tractable.

My Other Work on Learning and Experimentation

Several other papers of mine explore questions of experimentation and learning, although don’t use the Brownian motion framework. “Experimentation in Federal Systems” (with Bard Harstad, Quarterly Journal of Economics) looks at policy experimentation in federal systems and how those systems might be effectively designed to harness the power of experimentation. We began that project using the Brownian motion representation but ultimately decided the richness it provides was not necessary for our main point. The model very much leans on the key Brownian motion property of imperfect-invertibility but does so with a simplified reduced-form representation.

My recent working paper, “Regulating an Innovative Industry,” with Hongyi Li traversed a similar trajectory. Although inspired by models of cheap talk with complex expertise, we developed a model of multisender cheap talk when a regulator has to decide whether to approve a new innovation be brought to market. Although correlation of outcomes across different policy settings is more realistic, we ultimately developed the model for independent outcomes as it illuminated the driving mechanism more clearly. How Brownian uncertainty affects regulatory problems remains a fascinating and important question, and one we hope to address in future work.

A deliberate assumption in my Brownian motion papers is that the environment itself is unchanging yet difficult to learn. This naturally points to questions about how learning is effected when the environment itself changes. Two papers take up this question of change over time. “Gridlock and Delegation in a Changing World”, with Keith Krehbiel, and “Dynamic Policymaking with Decay”, with Greg Martin, both in the  American Journal of Political Science, explore how change affects the strategy of policymaking and the design of institutions.

Looking Ahead

There are many more open questions and interesting applications to tackle with the Brownian motion framework. I am actively working on some of these and am interested in exploring more. Please get in touch if you have any ideas or suggestions. Email me at sjc@stanford.edu. I hope you find the approach interesting!