Motivation and Overall Goal
Many data sets and real-world phenomena can be difficult for non-experts to understand. As a result, there is often a distance between useful knowledge and the audience, who may have difficulty in making correct inferences and potentially acting accordingly. The overall objective of Bivwac is to bridge this gap by studying new tools and methods that will help practitioners to better explain data and phenomena in various domains. Our main motivation is to contribute to the dissemination of knowledge in our societies, which can lead to better informed decisions and actions.
A likely reason for the difficulty in accessing knowledge is that in many cases the underlying data and phenomena can be very abstract or complex. For example, in the field of environmental issues, a lot of scientific data and predictive models exist, but there are still many misconceptions among the general public, and it is difficult for people to relate such information to their everyday experiences (Millot et al. 2018). This is also the case for mental health, where, for example, it can be difficult to develop a good understanding of disorders without experiencing what patients live every day (Violeau et al. 2020). As a result, many misconceptions, fears, and stigmas tend to exclude part of the world’s population. In terms of general education, many complex topics, such as emerging scientific disciplines (e.g., quantum physics), are reasonably well understood by experts but are still beyond the reach of most students. Again, this is largely due to the abstract and complex nature of the underlying phenomena, which makes them difficult to access with current tools. As a result, many students refrain from learning these topics, despite the need to educate new generations of students on these strategic issues (Bouchée et al. 2021.
In all of the examples discussed above, data and knowledge exist, but are underutilized and generally inaccessible to the general public. The Bivwac team explores how to make complex data and phenomena more accessible, understandable, and impactful by focusing on the design, implementation, and evaluation of immersive visualization experiences. Such visualization experiences have been under-researched. The content of these visualization experiences will be informed by experts from different domains, with whom we will collaborate to identify the main areas where they can benefit their domains and society in general. By identifying the factors that determine the success of immersive visualization experiences (such as the level of immersion or the degree of social interaction), and by creating new tools that help to transfer and promote useful knowledge, we hope to contribute to moving the world towards a more sustainable and collectively desirable future, in line with the United Nations Sustainable Development Goals.
What do we mean by Immersive Visualization Experiences?
By immersive visualization experience, we mean any computer-mediated or computer-enabled experience in which people can acquire new information while feeling immersed in it. This is in contrast to common situations in which a user learns new information while feeling somehow detached from it, as can be the case with regular information displays (e.g., looking at numbers or statistical graphs on a computer screen). Our working hypothesis is that in many cases immersion is associated with easier or more complete acquisition of information. We will now elaborate on what we mean by the three words immersive visualization experience, starting with the last one.
Experience. With the word experience, we want to emphasize that the process of exploring information and learning is a human activity and, as such, involves an experiential component. Thus, subjective experience will be central to our research. Drawing on the literature in psychology, we will explore how we can build on cognitive, affective, and conative aspects of experience (see, e.g., Hilgard, 1980). Beyond the simple presentation of raw data, we will explore how subjective experiences can help people build a more complete knowledge of complex mechanisms, data, and facts. For example, a course on the fundamentals of physics would be incomplete if it did not allow students to experience physical forces directly. As another example, information about an ongoing humanitarian crisis could be conveyed by showing data about the number and demographics of victims, or by showing videos that convey the suffering of some of the victims. It can be argued that either option would provide limited information, whereas a combination of the two could provide a much more complete picture (Dragicevic, 2022b. Thus, although our focus is on communicating data, facts, and evidence, we believe that more complete knowledge can often be conveyed by exploring more broadly how we can elicit holistic experiences that have educational value. Of course, even when presenting raw facts, human subjectivity cannot be avoided, as any way of presenting information will affect the way it is interpreted and create an experience that may involve emotions. Even the implications of data (e.g., learning the number of victims in a major earthquake or terrorist attack) can evoke emotions. Thus, it is important to scientifically study and understand the subjective and affective aspects of communicating information, even when the goal is to communicate facts and data as neutrally as possible.
Visualization. We use the word visualization very liberally; by visualization we mean, among other things, data visualizations (as studied in the areas of information visualization, scientific visualization, and visual analytics), but also any visual representation of information, be it quantitative or non-quantitative. Examples of visual representations of non-quantitative information include diagrams used to explain scientific concepts (e.g., Feynman diagrams), and photographs and videos meant to explain physical phenomena (e.g., photos of colored water to explain fluid mechanics). Compared to text and speech, images allow people to bypass the costly cognitive resources required for syntactic and semantic transcription ; This can free up their cognitive resources to fully engage in understanding, elaborating, and creating visual associations and configurations. Narratives are nonetheless powerful, so will extend our investigations to visual narratives and storytelling approaches (e.g., video tutorials or documentaries), and study ways they can be combined with factual approaches such as data visualization (Segel, 2010). Finally, we may even consider non-visual representations of information (e.g., haptic or auditory) (Prouzeau et al, 2019). To summarize, we are broadly interested in how computer technology can be used to make information more directly accessible to the human senses in ways that have not been previously explored or scientifically tested. Although using visual representations to convey information is far from being a new idea, a range of emerging technologies have appeared whose opportunities are only starting to be understood.
Immersive. We use the word immersive also very liberally. At a general level, immersion can refer to either perceptual immersion (e.g., being visually immersed in a 360° documentary) or cognitive immersion (e.g., reading a thrilling novel). We will be focusing on both types of immersion, and especially on ways the two can be combined, as both perceptual (Marriott et al, 2018) and cognitive (Isenberg et al, 2018) immersion can facilitate the comprehension of information. In terms of immersion, we will again be inclusive in our explorations and consider various input and output modalities including:
- Virtual reality (VR). VR is the most obvious form of immersion, and also the ones that have been the most explored in previous research. However, a novel stream of research called immersive analytics has recently emerged whose goal is to study how VR can be used to help people explore complex datasets in a way that is more engaging than regular visualizations (Marriott et al, 2018). We have been involved in this research (Ens et al., 2021), and we may continue this line of research, for example, by studying how immersive analytics can be complemented with an experiential component, an aspect that has been largely overlooked so far in this stream of research. Possible directions include designing data-driven experiences (Casamayou, 2022, and using VR to promote empathy and perspective-taking as has been done in previous work (Milk, 2015), but combined with data and facts (Dragicevic, 2022a).
- Augmented reality (AR). AR has received comparatively less attention than VR, but research attention has been growing recently. Compared to VR displays, AR displays make it possible to link the data directly to the surrounding physical environment, which often yields a richer perceptual experience and richer opportunities for action than pure VR setups. Thus situating or embedding virtual information into our physical environment (Willett et al., 2016) allows us to experience a physical immersion. In our past work, we have already explored how avatars in AR can be used for better perceiving the world around us (Genay. et al. 2022). In the future, we plan to study how AR can be used for better understanding data and phenomena anchored in the real world. This is for example the motivation of the Be·aware project regarding environmental issues where we test initial immersive experiences (Assor et al. 2022) - see Figure 1 and the dedicated section.
- Tangible user interfaces (TUIs). TUIs have a long research history, with studies suggesting that tangible objects can facilitate learning (O’Malley, 2004; Shaer, 2010). TUIs fit in our broad vision of immersive visualizations for two reasons: 1) they are typically more engaging than regular computer displays, and thus promote cognitive immersion; 2) they are often more deeply embedded in the physical world and as we have already discussed, the physical world is inherently immersive. In Potioc, we have extensively explored how to combine TUIs with spatial augmented reality (i.e., AR based on video projection) for educational purposes, see e.g. Teegi (Frey et al., 2014). In our future work, we will further push forward such approaches that remain underexplored for conveying complex knowledge.
- Physical visualizations. The study of physical visualizations (also called data physicalizations) is relatively new, and so far work has suggested that they can promote engagement and comprehension (Jansen et al., 2015). Two of the Bivwac members have spearheaded this stream of research, and we will continue to study it. For example, we have been recently interested in how to use participatory data physicalizations to help a small community learn about and exchange about the climate impact of their dietary choices (Sauvé, 2023). In the future, we will keep exploring how physical representations of information can convey data and phenomena.
All such user interface technologies (VR, AR, TUIs, physical visualizations) have different trade-offs that make them appropriate in some contexts and less in others. For example, VR isolates people from the physical environment while AR does not. Physical and tangible displays are persistent while digital displays rarely are. The existence of these trade-offs is why we need to study different types of technologies and examine ways in which they differ from each other.
For all these input and output modalities, we will study highly interactive experiences. Our objective is to encourage users to actively explore information, for example by leveraging the principles of explorable explanation to scrutinize explanations by interactively changing assumptions (Victor, 2011, Dragicevic et al, 2019. We will also put emphasis on collaborative experiences, which in most cases leads to better outcomes compared to individual learning (Dillenbourg et al., 2009). Various projects of Potioc have already started exploring collaborative tasks (e.g. Furio et al. 2017, Giraudeau et al., 2019). With the ANR project ICARE, and the PEPR eNSEMBLE network, we explore the design of techniques to improve collaboration between learners when using immersive technology.
Main application domains
Bivwac is interested in application domains where there exists data and knowledge that are difficult to comprehend by non-experts, and where better education is likely to bring positive societal consequences, as identified by the United Nation as Sustainable Development Goals (SDG). This includes the following families of application domains:
Environment (SDG #13 #14 #15)
We have identified environmental issues as a key domain we want to investigate, as rapid global action is required and any approach that may be able to mitigate the crisis deserves to be explored. We will be therefore looking at how immersive visualization experiences can be used to help practitioners better educate citizens, politicians, and decision makers for the promotion of pro-environmental decisions and actions as laid out by environmental sciences experts (e.g. IPPC) and government agencies such as ADEME. We will consider data and phenomena at different levels. This can be at a microscopic level where we focus on specific problems (e.g. understanding the relative importance of carbon footprint for different meals), or at a more macroscopic level where we try to convey a more global picture of available knowledge (e.g. better understanding the causes and consequences of global warming, the different levers of change, and the diverse implications of policies). We have already started to explore this direction of research with colleagues from CIRED, the international research center for ecology and development (see Be·aware project in Projects).
Education (SDG #4)
Beyond education on environmental issues, we are interested in education in general, in particular when knowledge is difficult to transmit with standard methods and user interfaces. As an example, in Potioc, we have worked on the teaching of wave optics for several years, and we have built a new interactive tool that goes beyond the limits of current teaching approaches (see Furio et al., 2017, and HOBIT webpage). In Bivwac, we will continue and extend this work in various areas, including quantum physics and collaborative learning (see Projects).
Global Welfare (SDG #3)
We will also be interested in how to promote global welfare. This domain is broad and will be broken down into several directions of research. As an example, we have started exploring how immersive AR visualizations can help people better understand what schizophrenia is, with the final goal of reducing stigma (see Projects). Other research directions relate to humanitarian visualizations where we want to study how immersive visualization experiences can be used to help reduce global suffering (Dragicevic, 2022a).
We chose to focus on domains that we currently think are the most likely to contribute to building a sustainable and collectively desirable world as laid out by the UN SDG, since those tend to be the most meaningful and the most inspiring to us. This focus is reflected by our choice of words in the Bivwac acronym (Welfare, Awareness, Comprehension). It will give a specific color and will contribute to make our project-team unique. For the application domains we target, the challenge will be to engage people in a process of learning, to promote a comprehensive understanding of poorly-understood phenomena, to encourage discussions and reflections between citizens or, potentially, to favor a real change in people’s behaviors. This differs from what is generally studied in more traditional contexts where productivity is often a target (optimize completion time, minimize error rate), which is not the case in our case. As a consequence, our work in Bivwac will explore new designs and new evaluation methodologies that stand out from the main research that is conducted in most of the other research groups (in particular in France).