ISMAR 2018 Tutorials

Augmented (AR) and Virtual Reality (VR) systems are designed to immerse humans into rich and compelling simulated environments by leveraging our perceptual systems (i.e. vision, touch, sound). In turn, exposure to AR/VR can reshape and alter our perceptual processing by tapping into the brain’s significant ability to adapt to changes in the environment through neural plasticity. Therefore, to design successful AR/VR systems, we must first understand the functioning and limitations of our perceptual and cognitive systems. We can then tailor AR/VR technology to optimally stimulate our senses and maximize user experience. Understanding how to wield AR/VR tools to reshape how we perceive the world also has incredible potential for societal and clinical applications.

By attending this tutorial, ISMAR attendees will learn how to employ our current understanding of human perception to design more sophisticated and ecological AR/VR systems that optimize user experience and minimize or even eliminate the undesired side effects. Additionally, AR/VR researchers attending this tutorial will learn how to exploit the brain’s neuroplasticity to reshape users’ conscious experiences in virtual environments. This knowledge could help evolve AR/VR systems from simple entertainment devices into tools to reshape human perception and behavior

In this tutorial, I will present the VR projects that I have worked on at the Amsterdam University of Applied Sciences, for our research project named ‘Storytelling for 360° Media’.

There are still many gaps in the knowledge on how to produce good VR content. Also, we actually don’t know that much yet on how audiences react to VR, and how we can measure concepts like presence, engagement and immersion.

In collaboration with industry partners and students, we have therefore set up a series of VR projects that were intended to explore the new film language that needs to be developed to fully benefit the possibilities of Cinematic VR. In every project, an element of storytelling has been explored. By reflecting on the results of our projects, I will share what we have learned about VR storytelling and how VR may be validated and evaluated.

This tutorial is not a static summing up of theoretical insights, but rather a discussion of a set of cases and issues that have led to insights into storytelling for VR. I will share our research questions, reflect on what went wrong - and eventually share the new questions that have arisen while doing it.

The rapid development of 3D scanning technology combined with state-of-the-art mapping algorithms allows to capture 3D point clouds with high resolution and accuracy. The high amount of data collected with LiDAR, RGB-D cameras or generated through SfM approaches makes the direct use of the recorded data for realistic rendering and simulation problematic. Therefore, these point clouds have to be transformed into representations that fulfill the computational requirements for VR and AR setups.

In this tutorial participants will be introduced to state-of-the-art methods in point cloud processing and surface reconstruction with open source software to learn the benefits for AR and VR applications by interleaved presentations, software demonstrations and software trials. The focus lies on 3D point cloud data structures (range images, octrees, k-d trees) and algorithms, and their implementation in C/C++. Surface reconstruction using Marching Cubes and other meshing methods will play another central role. Reference material for subtopics like 3D point cloud registration and SLAM, calibration, filtering, segmentation, meshing, and large scale surface reconstruction will be provided. Participants are invited to bring their Linux, MacOS or Windows laptops to gain hands-on experience on practical problems occuring when working with large scale 3D point clouds in VR and AR applications.

This tutorial will first discuss the replication crisis in empirical science. This term was coined to describe recent significant failures to replicate empirical findings in both medicine and psychology. In many cases, over 50% of previously reported results could not be replicated. This fact has shaken the foundations of both fields: Can empirical results really be believed? Should, for example, medical decisions really be based on empirical research?

After describing the crisis, the tutorial will revisit enough of the basics of empirical science to explain the origins of the replication crisis. The key issue is that hypothesis testing, which in empirical science is used to establish truth, is the result of a probabilistic process. However, the human mind is wired to reason absolutely: Humans have a difficult time understanding probabilistic reasoning*. The tutorial will discuss some of the ways that funding agencies, such as the US National Institutes of Health (NIH), have responded to the replication crisis, by, for example, funding replication studies, and requiring that grant recipients publically post anonymized data.

Finally, the tutorial will consider how the Virtual Environments community might respond to the replication crisis. In particular, in our community the reviewing process often considers work that involves systems, architectures, or algorithms. In these cases, the reasoning behind the correctness of the results is usually absolute. Therefore, the standard for accepting papers is that the finding exhibits novelty—to some degree, the result should be surprising. However, this standard does not work for empirical studies (which, typically, involve human experimental subjects). Because empirical reasoning is probabilistic, important results need to be replicated, sometimes multiple times, and by different laboratories. As the replications mount, the field is justified in embracing increasing belief in the results. In other words, consider a field that, in order to accept a paper reporting empirical results, always requires surprise: This is a field that will not progress in empirical knowledge.

The tutorial will end with a call for the community to be more accepting of replication studies. In addition, the tutorial will consider whether actions taken by other fields, in response to the replication crisis, might also be recommendable for the Virtual Environments community.