Uncertainty Visualization

Uncertainty is inherent to most data and can enter the analysis pipeline during the measurement, modeling, and forecasting phases. Effectively communicating uncertainty is necessary for establishing scientific transparency. Further, people commonly assume that there is uncertainty in data analysis, and they need to know the nature of the uncertainty to make informed decisions.

However, understanding even the most conventional communications of uncertainty is highly challenging for novices and experts alike, which is due in part to the abstract nature of probability and ineffective communication techniques. Reasoning with uncertainty is unilaterally difficult, but researchers are revealing how some types of visualizations can improve decision-making in a variety of diverse contexts,

  • from hazard forecasting,
  • to healthcare communication,
  • to everyday decisions about transit.

Scholars have distinguished different types of uncertainty, including

  • aleatoric (irreducible randomness inherent in a process),
  • epistemic (uncertainty from a lack of knowledge that could theoretically be reduced given more information),
  • and ontological uncertainty (uncertainty about how accurately the modeling describes reality, which can only be described subjectively).

The term risk is also used in some decision-making fields to refer to quantified forms of aleatoric and epistemic uncertainty, whereas uncertainty is reserved for potential error or bias that remains unquantified. Here we use the term uncertainty to refer to quantified uncertainty that can be visualized, most commonly a probability distribution. This article begins with a brief overview of the common uncertainty visualization techniques and then elaborates on the cognitive theories that describe how the approaches influence judgments. The goal is to provide readers with the necessary theoretical infrastructure to critically evaluate the various visualization techniques in the context of their own audience and design constraints. Importantly, there is no one-size-fits-all uncertainty visualization approach guaranteed to improve decisions in all domains, nor even guarantees that presenting uncertainty to readers will necessarily improve judgments or trust. Therefore, visualization designers must think carefully about each of their design choices or risk adding more confusion to an already difficult decision process.

Uncertainty Visualization Design Space

There are two broad categories of uncertainty visualization techniques. The first are graphical annotations that can be used to show properties of a distribution, such as the mean, confidence/credible intervals, and distributional moments.

Numerous visualization techniques use the composition of marks (i.e., geometric primitives, such as dots, lines, and icons) to display uncertainty directly, as in error bars depicting confidence or credible intervals. Other approaches use marks to display uncertainty implicitly as an inherent property of the visualization. For example, hypothetical outcome plots (HOPs) are random draws from a distribution that are presented in an animated sequence, allowing viewers to form an intuitive impression of the uncertainty as they watch.

The second category of techniques focuses on mapping probability or confidence to a visual encoding channel. Visual encoding channels define the appearance of marks using controls such as color, position, and transparency. Techniques that use encoding channels have the added benefit of adjusting a mark that is already in use, such as making a mark more transparent if the uncertainty is high. Marks and encodings that both communicate uncertainty can be combined to create hybrid approaches, such as in contour box plots and probability density and interval plots.

More expressive visualizations provide a fuller picture of the data by depicting more properties, such as the nature of the distribution and outliers, which can be lost with intervals. Other work proposes that showing distributional information in a frequency format (e.g., 1 out of 10 rather than 10%) more naturally matches how people think about uncertainty and can improve performance.

Visualizations that represent frequencies tend to be highly effective communication tools, particularly for individuals with low numeracy (e.g., inability to work with numbers), and can help people overcome various decision-making biases.

Researchers have dedicated a significant amount of work to examining which visual encodings are most appropriate for communicating uncertainty, notably in geographic information systems and cartography. One goal of these approaches is to evoke a sensation of uncertainty, for example, using fuzziness, fogginess, or blur.

Other work that examines uncertainty encodings also seeks to make looking-up values more difficult when the uncertainty is high, such as value-suppressing color pallets.

Given that there is no one-size-fits-all technique, in the following sections, we detail the emerging cognitive theories that describe how and why each visualization technique functions.

VU1

Uncertainty Visualization Theories

The empirical evaluation of uncertainty visualizations is challenging. Many user experience goals (e.g., memorability, engagement, and enjoyment) and performance metrics (e.g., speed, accuracy, and cognitive load) can be considered when evaluating uncertainty visualizations. Beyond identifying the metrics of evaluation, even the most simple tasks have countless configurations. As a result, it is hard for any single study to sufficiently test the effects of a visualization to ensure that it is appropriate to use in all cases. Visualization guidelines based on a single or small set of studies are potentially incomplete. Theories can help bridge the gap between visualizations studies by identifying and synthesizing converging evidence, with the goal of helping scientists make predictions about how a visualization will be used. Understanding foundational theoretical frameworks will empower designers to think critically about the design constraints in their work and generate optimal solutions for their unique applications. The theories detailed in the next sections are only those that have mounting support from numerous evidence-based studies in various contexts. As an overview, The table provides a summary of the dominant theories in uncertainty visualization, along with proposed visualization techniques.

UV2

General Discussion

There are no one-size-fits-all uncertainty visualization approaches, which is why visualization designers must think carefully about each of their design choices or risk adding more confusion to an already difficult decision process. This article overviews many of the common uncertainty visualization techniques and the cognitive theory that describes how and why they function, to help designers think critically about their design choices. We focused on the uncertainty visualization methods and cognitive theories that have received the most support from converging measures (e.g., the practice of testing hypotheses in multiple ways), but there are many approaches not covered in this article that will likely prove to be exceptional visualization techniques in the future.

There is no single visualization technique we endorse, but there are some that should be critically considered before employing them. Intervals, such as error bars and the Cone of Uncertainty, can be particularly challenging for viewers. If a designer needs to show an interval, we also recommend displaying information that is more representative, such as a scatterplot, violin plot, gradient plot, ensemble plot, quantile dotplot, or HOP. Just showing an interval alone could lead people to conceptualize the data as categorical. As alluded to in the prior paragraph, combining various uncertainty visualization approaches may be a way to overcome issues with one technique or get the best of both worlds. For example, each animated draw in a hypothetical outcome plot could leave a trace that slowly builds into a static display such as a gradient plot, or animated draws could be used to help explain the creation of a static technique such as a density plot, error bar, or quantile dotplot. Media outlets such as the New York Times have presented animated dots in a simulation to show inequalities in wealth distribution due to race. More research is needed to understand if and how various uncertainty visualization techniques function together. It is possible that combining techniques is useful in some cases, but new and undocumented issues may arise when approaches are combined.

In closing, we stress the importance of empirically testing each uncertainty visualization approach. As noted in numerous papers, the way that people reason with uncertainty is non-intuitive, which can be exacerbated when uncertainty information is communicated visually. Evaluating uncertainty visualizations can also be challenging, but it is necessary to ensure that people correctly interpret a display. A recent survey of uncertainty visualization evaluations offers practical guidance on how to test uncertainty visualization techniques.

Click her to access the entire article in Handbook of Computational Statistics and Data Science

An Animal Kingdom Of Disruptive Risks -How boards can oversee black swans, gray rhinos, and white elephants

Where was the board? As a corporate director, imagine you find yourself in one of these difficult situations:

  • Unexpected financial losses mount as your bank faces a sudden collapse during a 1-in-100-year economic crisis.
  • Customers leave and profits drop year after year as a new technology start-up takes over your No. 1 market position.
  • Negative headlines and regulatory actions besiege your company following undesirable tweets and other belligerent behavior from the CEO.

These scenarios are not hard to imagine when you consider what unfolded before the boards of Lehman Brothers, Blockbuster, Tesla, and others. In the context of disruptive risks, these events can be referred to as black swans, gray rhinos, and white elephants, respectively. While each has unique characteristics, the commonality is that all of these risks can have a major impact on a company’s profitability, competitive position, and reputation.

In a VUCA (volatile, uncertain, complex, and ambiguous) world, boards need to expand their risk governance and oversight to include disruptive risks. This article addresses three fundamental questions:

  • What are black swans, gray rhinos, and white elephants?
  • Why are they so complex and difficult to deal with?
  • How should directors incorporate these disruptive risks as part of their oversight?

Why are companies so ill prepared for disruptive risks? There are three main challenges:

  1. standard enterprise risk management (ERM) programs may not capture them;
  2. they each present unique characteristics and complexities;
  3. and cognitive biases prevent directors and executives from addressing them.

Standard tools used in ERM, including risk assessments and heat maps, are not timely or dynamic enough to capture unconventional and atypical risks. Most risk quantification models—such as earnings volatility and value-at-risk models—measure potential loss within a 95 percent or 99 percent confidence level. Black swan events, on the other hand, may have a much smaller than 0.1 percent chance of happening. Gray rhinos and white elephants are atypical risks that may have no historical precedent or operational playbooks. As such, disruptive risks may not be adequately addressed in standard ERM programs even if they have the potential to destroy the company. The characteristics and complexities of each type of disruptive risk are unique. The key challenge with black swans is prediction. They are outliers that were previously unthinkable. That is not the case with gray rhinos, since they are generally observable trends. With gray rhinos the main culprit is inertia: companies see the megatrends charging at them, but they can’t seem to mitigate the risk or seize the opportunity. The key issue with white elephants is subjectivity. These no-win situations are often highly charged with emotions and conflicts. Doing nothing is usually the easiest choice but leads to the worst possible outcome. While it is imperative to respond to disruptive risks, cognitive biases can lead to systematic errors in decision making. Behavioral economists have identified dozens of biases, but several are especially pertinent in dealing with disruptive risks:

  • Availability and hindsight bias is the underestimation of risks that we have not experienced and the overestimation of risks that we have. This bias is a key barrier to acknowledging atypical risks until it is too late.
  • Optimism bias is a tendency to overestimate the likelihood of positive outcomes and to underestimate the likelihood of negative outcomes. This is a general issue for risk management, but it is especially problematic in navigating disruptive risks.
  • Confirmation bias is the preference for information that is consistent with one’s own beliefs. This behavior prevents us from processing new and contradictory information, or from responding to early signals.
  • Groupthink or herding occurs when individuals strive for group consensus at the cost of objective assessment of alternative viewpoints. This is related to the sense of safety in being part of a larger group, regardless if their actions are rational or not.
  • Myopia or short-termism is the tendency to have a narrow view of risks and a focus on short-term results (e.g., quarterly earnings), resulting in a reluctance to invest for the longer term.
  • Status quo bias is a preference to preserve the current state. This powerful bias creates inertia and stands in the way of appropriate actions.

To overcome cognitive biases, directors must recognize that they exist and consider how they impact decision making. Moreover,

  • board diversity,
  • objective data,
  • and access to independent experts

can counter cognitive biases in the boardroom.

Recommendations for Consideration

How should directors help their organizations navigate disruptive risks? They can start by asking the right questions in the context of the organization’s business model and strategy. The chart below lists 10 questions that directors can ask themselves and management.

NACD1

In addition, directors should consider the following five recommendations to enhance their risk governance and oversight:

  1. Incorporate disruptive risks into the board agenda. The full board should discuss the potential impact of disruptive risks as part of its review of the organization’s strategy to create sustainable long-term value. Disruptive risks may also appear on the agenda of key committees, including the risk committee’s assessment of enterprise risks, the audit committee’s review of risk disclosures, the compensation committee’s determination of executive incentive plans, and the governance committee’s processes for addressing undesirable executive behavior. The key is to explicitly incorporate disruptive risks into the board’s oversight and scope of work.
  2. Ensure that fundamental ERM practices are effective. Fundamental ERM practices—risk policy and analytics, management strategies, and metrics and reporting—provide the baseline from which disruptive risks can be considered. As an example, the definition of risk appetite can inform discussions of loss tolerance relative to disruptive risks. As an early step, the board should ensure that the overall ERM framework is robust and effective. Otherwise, the organization may fall victim to “managing risk by silo” and miss critical interdependencies between disruptive risks and other enterprise risks.
  3. Consider scenario planning and analysis. Directors should recognize that basic ERM tools may not fully capture disruptive risks. They should consider advocating for, and participating in, scenario planning and analysis. This is akin to tabletop exercises for cyber-risk events, except much broader in scope. Scenario analysis can be a valuable tool to help companies put a spotlight on hidden risks, generate strategic insights on performance drivers, and identify appropriate actions for disruptive trends. The objective is not to predict the future, but to identify the key assumptions and sensitivities in the company’s business model and strategy. In addition to scenario planning, dynamic simulation models and stress-testing exercises should be considered.
  4. Ensure board-level risk metrics and reports are effective. The quality of risk reports is key to the effectiveness of board risk oversight. Standard board risk reports often are comprised of insufficient information: historical loss and event data, qualitative risk assessments, and static heat maps. An effective board risk report should include quantitative analyses of risk impacts to earnings and value, key risk metrics measured against risk appetite, and forwardlooking information on emerging risks. By leveraging scenario planning, the following reporting components can enhance disruptive risk monitoring:
    • Market intelligence data that provides directors with useful “outside-in” information, including key business and industry developments, consumer and technology trends, competitive actions, and regulatory updates.
    • Enterprise performance and risk analysis including key performance and risk indicators that quantify the organization’s sensitivities to disruptive risks.
    • Geo-mapping that highlights global “hot spots” for economic, political, regulatory, and social instability. This can also show company-specific risks such as third-party vendor, supply chain, and cybersecurity issues.
    • Early-warning indicators that provide general or scenariospecific signals with respect to risk levels, effectiveness of controls, and external drivers.
    • Action triggers and plans to facilitate timely discussions and decisions in response to disruptive risks.
  5. Strengthen board culture and governance. To effectively oversee disruptive risks, the board must be fit for purpose. This requires creating a board culture that considers nontraditional views, questions key assumptions, and supports continuous improvement. Good governance practices should be in place in the event a white elephant appears. For example, what is the board protocol and playbook if the CEO acts inappropriately? In the United States, the 25th Amendment and impeachment clauses are in place ostensibly to remove a reprehensible president. Does the organization have procedures to remove a reprehensible CEO?

The following chart summarizes the key characteristics, examples, indicators, and strategies for identifying and addressing black swans, gray rhinos, and white elephants. The end goal should be to enhance oversight of disruptive risks and counter the specific challenges that are presented. To mitigate the unpredictability of black swans, the company should develop contingency plans with a focus on preparedness. To overcome inertia and deal with gray rhinos, the company needs to establish organizational processes and incentives to increase agility. To balance subjectivity and confront white elephants, directors should invest in good governance and objective input that will support decisiveness.

NACD2

The Opportunity for Boards

In a VUCA world, corporate directors must expand their traditional risk oversight beyond well-defined strategic, operational, and financial risks. They must consider atypical risks that are hard to predict, easy to ignore, and difficult to address. While black swans, gray rhinos, and white elephants may sound like exotic events, directors could enhance their recognization of them by reflecting on their own experiences serving on boards.

Given their experiences, directors should provide a leading voice to improve oversight of disruptive risks. They have a comparative advantage in seeing the big picture based on the nature of their work— part time, detached from day-to-day operations, and with experience gained from serving different companies and industries. Directors can add significant value by providing guidance to management and helping them see the forest for the trees. Finally, there is an opportunity side to risk. There are positive and negative black swans. A company can invest in the positive ones and be prepared for the negative ones. For every company that is trampled by a gray rhino, another company is riding it to a higher level of performance. By addressing the white elephant in the boardroom, a company can remediate an unspoken but serious problem. In the current environment, board oversight of disruptive risks represents both a risk management imperative and a strategic business opportunity.

Click here to access NACD’s summary