Modeling Hard Problems: Learning in a Complicated World

In many settings agents face difficult problems, be they managers, policymakers, judges, or consumers. A significant part of the difficulty is that decision makers face many options from which to choose yet possess only a tenuous understanding of how these alternatives map into outcomes.  I have introduced an approach to modeling problems of this sort and, along with co-authors, explored its application to a variety of settings in economics, political science, and management.

The key novelty of my approach is to suppose that the mapping from actions to outcomes is the realized path of a Brownian motion. The mapping is fixed throughout time and the agents can learn about the mapping by sampling points and observing the outcomes. As time passes, therefore, agents learn more and more points in the mapping, developing a better understanding of the environment in which they operate but, crucially, never knowing it completely. 

The variance of the Brownian motion captures the complexity of the decision making environment. The information space is rich—there is a continuum of state variables (one for each action)—but the states are correlated. This correlation allows agents to learn from their experience as information spills-over from one alternative to others. The difficulty in learning about an environment depends on the degree of this correlation. Thus, the variance of the Brownian motion—how correlated are nearby alternatives–measures the complexity of the decision making environment.

The approach represents a non-standard use of the Brownian motion. In particular, the Brownian motion does not evolve through time. (One may think of the process as having evolved through the action space at time 0 before play). I leverage the mathematics of the Brownian motion to provide a concise and rich representation of uncertainty and learning in complicated environments.

I have used the Brownian motion representative to explore a variety of problems, touching upon applications otherwise considered distinct. Unifying these applications with a common model of uncertainty is an objective of my work. I developed this unified perspective in a short course I presented in 2020. Details and slides are below.

1. Experimentation and Dynamic Learning

A starting point is the classic question of experimentation: When do agents seek safety in repeating past (and known) actions versus experimenting with a risky but potentially superior action? This is formalized as the classic bandit problem in which there is a set of discrete and independent (uncorrelated) arms. The Brownian motion approach allows for the arms to be correlated and for there to be a continuum of them that are deterministic. Thus, the question is not “how long” to pursue an arm, where the arms are independent and essentially indistinguishable. Rather, the question is which arm should I choose given what I have learned so far?

The Brownian motion approach is appealing also as it offers a notion of distance. How “far” is an experiment from what is known? Thus, the approach opens up new questions, such as How bold is the experiment when it does happen and in which direction do agents search? Does learning ultimately converge on good alternatives or can it get stuck at inefficient action? How is learning affected, be it distorted or improved, when agents have different preferences?

Experimentation & Learning:

In “Searching and Learning by Trial and Error,” I explore this model of experimentation and learning in the context of product markets. I characterize the equilibrium search path and show how it falls into two natural phases before stabilizing. I show how search can return to a previously discarded alternative and get “stuck” in either phase, leading to significant inefficiency.  I then show how both of these results can explain the famed “product life cycle” in markets.

An appeal of the Brownian motion approach is that, in many ways, it is highly tractable. The richness of uncertainty does come with some trade-offs, however. One  such trade-off is that it is difficult is to capture forward looking behavior. In the above paper I assume agents are myopic. 

In “Preemptive Policy Experimentation,” we extend the agents’ planning horizon to two periods and show how the long shadow of policy experimentation can significantly shape current policy choices when policymakers have different objectives. (See below for more on forward-looking behavior.)

An assumption in both of these papers is that agents possess an ideal outcome and they search for an alternative that gets as close as possible to this outcome. In many economic environments, agents are never satiated and always seek more (higher and higher performance, for example). In “The Risk of Failure,” we adapt the search model to the allow for these preferences. In this environment search is constrained only by risk aversion, and we examine the impact of risk on experimentation and what it means for long-run performance.

In a (long-percolating) working paper with Niko Matouschek, “Managing on Rugged Landscapes,” we apply the Brownian model to learning within firms, exploring managerial search, imitation, and coordination.

Experimentation & Learning in Policymaking and Competitive Elections:

How do complex environments affect strategic behavior when agents have different preferences over the outcomes produced? I explore these questions in the domain of political economy through the classic lens of political parties competing over policy.

In “Searching for Good Policies” I show how competitive elections can facilitate more effective experimentation and produce better long-term outcomes for voters, even when those parties are partisan with different preferences to voters.

In “Preemptive Policy Experimentation” we show how the likelihood of being succeeded in office by a party with different preferences can induce policy experimentation where it otherwise wouldn’t occur. The early policymaker, in a sense, experiments so as to teach his successor about the policy mapping, hoping to shape his successor’s policy choice. This result shows how policy choices can cast long shadows over the future even when the linkage across time is purely informational.

In “Agenda Control Under Policy Uncertainty,” we apply the Brownian structure to Romer and Rosenthal’s classic agenda setter model. We show how Brownian policy uncertainty upends many of Romer and Rosenthal’s classic insights and expands the classic gridlock interval.

A common element to all papers listed here is that the only linkage across time is informational. In Section 4 below, I describe recent work on competition in markets that relaxes this assumption. (This feature is also present in an extension in “Agenda Control Under Policy Uncertainty;” see the discussion below.)

Other researchers. Several other researchers have adopted this approach as well and have made progress on the challenge of forward-looking agents. Bruno Strulovici and Umberto Garfagnini use the Brownian motion structure in their paper, “Social Experimentation with Interdependent and Expanding Technologies” (Review of Economic Studies, 2016) to study an overlapping generations of agents where each generation has a two-period planning horizon. They show how this can generate waves of innovation through time.

Significant progress has been made in recent work by Can Urgun and Leeat Yariv (Princeton), Yu Fu Wong (Columbia) and Shangen Li (Zurich). In various ways, they allow for fully-forward looking agents by restricting attention to continuous search over the Brownian path.

2. Strategic Communication

The Brownian motion can be applied to questions of strategic communication. Information is asymmetric with the expert knowing the state of the world. In contrast to the canonical models in which expertise is a single piece of information (Crawford and Sobel 1982; Milgrom 1981), the expert knows the full path of the Brownian motion. In this line of work I study these “complex environments” and how strategic communication between an expert and a decision maker can change relative to the simple expertise of the canonical models.  These models are one-shot rather than dynamic. 

Cheap Talk:

In these papers I study the canonical model of cheap talk of Crawford and Sobel (1982) when expertise is complex. We show how communication can be efficient — with no wasted information — and sender-optimal when expertise is complex. Both of these properties contrast with the partitional equilibria of simple environments. The key insight is that in complex environments experts are able to communicate precisely yet imperfectly. Because the expert’s informational advantage is large, she can recommend her most preferred action without giving up all of her information to the receiver. The expert can use her information and at the same time keep some of it private. We identify a strategy by which the sender can do this and control the spillover of information so that she can retain leverage over the decision. Indeed, we show that the equilibrium outcome is the same as it would be if the expert held full decision making authority.

The working paper with Yunus Aybas, a PhD student at Stanford, provides a characterization of a sender-optimal equilibria for small and large bias, showing how it depends on the size of the action space and the nature of the informational spillover from the sender’s recommendation. By identifying the key ingredients to sustain efficient cheap talk, it extends beyond the Brownian motion environment and describes other complex environments in which efficient cheap talk is possible. 

 The earlier paper, “A Theory of Policy Expertise,” was the first paper I wrote using the Brownian motion framework. It identifies the equilibrium for small bias only and only when the action space is unbounded. It then focuses on how the different nature of advice creates an incentive for the sender to acquire expertise, an incentive that isn’t present in the same way in the traditional Crawford-Sobel setting. Interestingly, the agent who has the most incentive to acquire expertise is an agent with positive bias (thus, different outcome preferences to the decision maker). This shows why strategic communication may often be between players with divergent preferences.

Hard (Verifiable) Information:

We show that in complex environments, the nature of communication qualitatively changes. With information verifiable, experts now provide advice that is more than just a recommendation. They provide “referential” as well as “conative” advice. That, unlike existing models, experts communicate additional information about the environment in addition to a recommendation. We show that by providing this additional information, and only by providing this additional information, is the expert able to leverage her expertise and sway decisions to her own advantage.

Other Equilibria. A limitation of my work in this area is that we characterize only some equilibria, focusing on those that deliver leverage to the expert. A more complete exploration of equilibria, particularly in the cheap talk setting, is a natural and needed direction for follow-on work. Identifying other equilibria is important to understanding the nature of strategic communication in complex environments. Additionally, it will help create a bridge between equilibria in the simple environments of the literature and complex environments that I have studied.

Dynamics. That experimentation and strategic communication are built on the same informational structure highlights the connections between two fields that have otherwise been treated as distinct. In work-in-progress, I am exploring dynamic models that combine strategic communication and experimentation.

3. Strategic Learning when Facing a Flow of Problems

Common to the experimentation and communication papers is that the decision maker seeks a single action that delivers the best outcome to maximize her utility. Any other learning is only incidental and a means-to-this-end. In many decision making environments, the players care about the outcome of many actions, possibly even the entire function. I explore this type of problem in the paper:

This paper considers the problem of courts facing a flow of related cases on which verdicts must be rendered. The cases are related but differ ever so slightly or significantly from each other. As much as we’d like to think of judges handing down justice impartially and without error, the reality is much different. Judges must issue judgements with incomplete information and imperfect precedents to guide their thinking.

We use the Brownian motion to capture this problem faced by the courts. Each point in the real line corresponds to a set of case facts that can appear in front of a judge. The judge must determine the correct judgement corresponding to each set of case facts, whether that be guilty or innocent, liable or not liable, etc. The problem is that the courts do not know the mapping of real-world facts into legal outcomes. We suppose a standard judicial hierarchy, and that the Higher Court can learn the legal outcome for any case, but that it has limited resources and can hear only a single case in each period. The flow of cases–a continuum, we assume–must be heard by the Lower Courts, who cannot discern correct legal outcomes, but do have available the precedents issued by the Higher Court. The questions we explore are: (1) How do the Lower Courts adjudicate in such an informationally complex environment? (2) How does this equate with the common, but much criticized, practice of analogical reasoning? (3) What does these answers imply for how the Higher Court chooses to hear cases and how it shapes its doctrine to guide Lower Court decision making?

Other researchers. Our understanding of decision making incentives in this type of problem has been advanced considerably by other researchers. See the work of Arjada Bardhi (Duke) in a paper on “Attributes” (R&R at Econometrica) and in a joint paper with Nina Bobkova (Rice) on “Local Evidence and Diversity in Minipublics” (forthcoming at the JPE), as well as an exciting paper on how scientists “stand on the shoulders of giants” — titled “A Quest for Knowledge” –by Johannes Schneider (UC3) and Christoph Carnehl (Bocconi).

4. Market Competition

Innovation is integral to economic growth. A long standing question in the Industrial Organization (IO) literature is the extent to which market structure incentivizes innovation. By its very nature, innovation is speculative and uncertain, with many new technologies, products, and potential markets to try. In recent work, we have begun to apply the Brownian motion framework to this problem to understand how the difficulty in finding successful innovations shapes the strategy of firms and the outcomes of market competition. 

In “The Novelty of Innovation,” we develop a simple model of entry by a new firm. In using the Brownian motion to represent the quality of potential innovations, we are able to capture a continuous notion of distance between technologies that has not appeared in the literature. This allows us to get at the question of not whether innovation occurs but what innovations are pursued and succeed in the market. We explore the implications of the model for the application of antitrust policy in innovative industries.

In “Innovation and Competition on a Rugged Technological Landscape,” we explore this model dynamically. In each period a new firm decides whether to enter the market and, if so, where to locate. For the innovation decision of each firm, this provides a rich choice. In addition to a bold or an incremental innovation, the firm decides whether to innovate in a niche, within the boundaries of the extant market, or beyond the frontier, expanding the scope of the market. We characterize the innovation strategies of the individual firms and the dynamic path and efficiency of the market over time.

Our AEA Papers & Proceedings paper summarizes and expands upon the ideas in both of these papers in a concise form.

The key technical novelty of these papers is that the linkage across periods is no longer purely informational. The payoff a firm receives depends on the outcome of the mapping at their location and also the location of the other firm(s) that it must compete with. In this way, innovation and competition interact and shape the market strategies of the firms. The firms are no longer just searching, they are searching and competing simultaneously.

As mentioned above, the political economy paper “Agenda Control under Policy Uncertainty” contains a two-period extension that also has this feature. In that context, the linkage across time is that the policy choice today becomes the status quo tomorrow and, given the collective choice requirement, this shapes future policymaking possibilities. This connects to the political economy literature on an endogenous status quo.

Learning Across Alternatives — A Short Course

In October 2020, I delivered a short course on my work in this area. The title of the course was “Learning Across Alternatives: Strategic Behavior in a Complicated World.” The short course was delivered in conjunction with the Org Econ Workshop organized annually in Sydney (a great conference! You should consider attending in the future). The course was broken down along the lines of the four decision problems listed above. The slides for the lectures are available here:

Formal Structure of the Model

Formally, the Brownian motion representation corresponds to a bandit problem with a continuum of correlated, deterministic bandits. The key feature is the correlation. The classic assumption in the experimentation literature is of independent bandits, typically with stochastic outcomes (and usually with a single, or at most finite, risky arm). The question the literature explores is how long to persist with an arm before abandoning it. Notably, when an arm is abandoned, nothing new is learned about the other arms and search begins again from scratch. With correlation across the arms this property is no longer true. In the Brownian framework, every experiment – be it a success or a failure – informs future choice. A success guides the way to where good alternatives are to be found, and a failure the opposite. This interdependence of outcomes leads to rich and path dependent behavior. Yet it does come (for the moment at least) with an analytic cost. In most of my search papers I assume agents are short-sighted, optimizing utility on a per-period basis. The exceptions are “Preemptive Policy Experimentation” that extends the horizon to two periods, and “The Risk of Failure,” that shows the structure of myopic search is robust to far-sighted agents. It is well known from the bandit literature that correlation is difficult to handle analytically. Nevertheless, the Brownian framework provides enough structure on beliefs (particularly through the Markov property) that the model with far-sighted behavior may yet prove tractable.

My Other Work on Learning and Experimentation

Several other papers of mine explore questions of experimentation and learning, although don’t use the Brownian motion framework. “Experimentation in Federal Systems” (with Bard Harstad, Quarterly Journal of Economics) looks at policy experimentation in federal systems and how those systems might be effectively designed to harness the power of experimentation. We began that project using the Brownian motion representation but ultimately decided the richness it provides was not necessary for our main point. The model very much leans on the key Brownian motion property of imperfect-invertibility but does so with a simplified reduced-form representation.

A deliberate assumption in my Brownian motion papers is that the environment itself is unchanging yet difficult to learn. This naturally points to questions about how learning is effected when the environment itself changes. Two papers take up this question of change over time. “Gridlock and Delegation in a Changing World”, with Keith Krehbiel, and “Dynamic Policymaking with Decay”, with Greg Martin, both in the  American Journal of Political Science, explore how change affects the strategy of policymaking and the design of institutions.

Looking Ahead

There are many more open questions and interesting applications to tackle with the Brownian motion framework. I am actively working on some of these and am interested in exploring more. Please get in touch if you have any ideas or suggestions. Email me at sjc@stanford.edu. I hope you find the approach interesting!