ISMAR Posters

Estimation of 6DoF object poses from single images is a problem of great interest in augmented reality and robotics research since it enables interaction with the object or initialization of pose tracking. Approaches utilizing deep neural networks have shown good performance, however the majority of them rely on training on real images of the objects which can be challenging in terms of ground truth pose acquisition, scalability and full coverage of possible poses. In this paper, we disregard all depth and color information and train a CNN to directly regress 6DoF object poses using only synthetic single channel edge enhanced images. We evaluate our approach against the state-of-the-art using synthetic training images and show a significant improvement on the commonly used LINEMOD benchmark dataset.

Existing camera-projector calibration methods typically warp feature points from a camera image to a projector image using estimated homographies, and often suffer from errors in camera parameters and noise due to imperfect planarity of the calibration target. In this paper we propose a simple yet robust solution that explicitly deals with these challenges. Following the structured light (SL) camera-project calibration framework, a carefully designed correspondence algorithm is built on top of the De Bruijn patterns. Such correspondence is then used for initial camera-projector calibration. Then, to gain more robustness against noises, especially those from an imperfect planar calibration board, a bundle adjustment algorithm is developed to jointly optimize the estimated camera and projector models. Aside from the robustness, our solution requires only one shot of SL pattern for each calibration board pose, which is much more convenient than multi-shot solutions in practice. Data validations are conducted on both synthetic and real datasets, and our method shows clear advantages over existing methods in all experiments.

We present a dynamic omnidirectional texture synthesis (DOTS) approach for generating real-time virtual reality content captured using a consumer-grade RGB-D camera. Compared to a single fixed-viewpoint color map, view-dependent texture mapping (VDTM) techniques can reproduce finer detail and replicate dynamic lighting effects that become especially noticeable with head tracking in virtual reality. However, VDTM is very sensitive to errors such as missing data or inaccurate camera pose estimation, both of which are commonplace for objects captured using consumer-grade RGB-D cameras. To overcome these limitations, our proposed optimization can synthesize a high-resolution view-dependent texture map for any virtual camera location. Synthetic textures are generated by uniformly sampling a spherical virtual camera set surrounding the virtual object, thereby enabling efficient real-time rendering for all potential viewing directions.

State of the art methods for diminished reality (DR) propagate pixel information from keyframes to later frames to achieve real-time image inpainting in 3D spaces. However, this approach assumes a planar scene and produces artifacts, if the scene geometry is not sufficiently planar. In this paper, we present 3D PixMix, a new real-time inpainting method that addresses non-planar scenes by considering both color and depth information in the inpainting process. We define cost functions for both the color and the geometric appearance in the inpainting scheme and use an RGB-D sensor for depth fusion using a SLAM. Comparison results against the conventional PixMix show that 3D PixMix obtains the equivalent or even better quality in 3D scenes with additional depth information.

In this paper; we want to introduce a novel mixed reality (MR) analysis tool that provides 3D reproductions of multiple actors using head-mounted AR devices in a cooperative setting. The tool is motivated by a challenge we are faced with when analysing AR support for cooperative scenarios: For this analysis; one needs information on people's movements and behaviour as well as on their interaction with digital information and objects. As other means of analysis proved insufficient for these purposes; we created and applied a novel tool which shows wearers of Head Mounted Displays (HMD) devices; the digital objects they interacted with and the actual setting they were in. The tool features several features for the analysis of these settings; which were predominately based on requirements stemming from two cases studies. In this paper; we present the tool; its most prominent features; and its application in two cases. To our knowledge; there is no other tool that includes similar features and means for analysis available; and we would like to discuss its benefits and possible applications with interested colleagues.

This paper presents an approach to semantic segmentation and structural modeling from dense 3D point clouds. The core contribution is an efficient method for fitting of geometric primitives based on machine learning. First, the dense 3D point cloud is acquired together with RGB images on a mobile handheld device. Then, RANSAC is used to estimate the presence of geometric primitives, followed by an evaluation of their fit based on classification of the fitting parameters. Finally, the approach iterates over successive frames to optimize the fitting parameters or replace a detected primitive by a better fitting one. As a result, we obtain a semantic model of the scene consisting of a set of geometric primitives. We evaluate the approach on an extensive set of scenarios and show its plausibility in augmented reality applications.

Realistic augmented reality systems require both accurate localization of the user and a mapping of the environment. In a markerless environment this is often done with SLAM algorithms which, for localization, pick out features in the environment and compare how they have changed from keyframe to current frame. However, human head agility, such as seen in video gaming tasks or training exercises, poses a problem; fast rotations will cause all previously tracked features to no longer be within the field of view and the system will struggle to localize accurately. In this paper we present an approach that is capable of tracking a human head's agile movements by using an array of RGB-D sensors and a reconstruction of this sensor data into 360 degrees of features that is fed into our SLAM algorithm. We run an experiment with pre-recorded agile movement scenarios that demonstrate the accuracy of our system. We also compare our approach against single sensor algorithms and show a significant improvement (up to 15 to 20 times better accuracy) in localization. The development of our sensor array and SLAM algorithm creates a novel approach to accurately localize extremely agile human head movements.

Detecting moving objects in the real world with reliability, robustness and efficiency is an essential but difficult task in AR applications, especially for interactions between virtual agents and real pedestrians, motorcycles and more, where the spatial occupancy of non-rigid objects should be perceived. In this paper, a novel object tracking method using visual cues with pre-training isproposed to track dynamic objects in 2D online videos robustly and reliably. The object's area in images can be transformed to 3D spatial area in the physical world with some simple, well-defined constraints and priors, thus spatial collision of agents and pedestrians can be avoided in AR environments. To achieve robust tracking in a markerless AR environment, we first create a novel representation of non-rigid objects, which is actually the manifold of normalized sub-images of all the possible appearances of the target object. These sub-images, captured from multiple views and under varying lighting conditions, are free from any occlusion and can be obtained from both video sequences and synthetic image generation. Then, from the instance pool made up of these sub-images, a compact set of templates which can well represent the manifold is learned by our proposed iterative method using sparse dictionary learning. We ensure that this template set is complete by using an SVM-based sparsity detection method. This compact, complete set of templates is then used to track the target trajectory online in video and augmented reality (AR) systems. Experiments demonstrate the robustness and efficiency of our method.

A projector camera system for controlling the appearance of real objects is useful for public viewing. However, projector camera systems need to estimate scenes under the white light from a camera image taken under the color light of the projector and adaptively generate a new projector image. In such a feedback system, the quality of the estimation of the scene color under the white light affects the quality of the projection image directly. In this study, we propose a precise online estimation of the scene color assuming that the reflectance matrix non-diagonal and can be determined with three parameters. This enables the projector-camera system to be applied to design tools or food AR, which require a highquality grade for material perception simulations.

Depth map super-resolution is a challenging computer vision problem. In this paper, we present two deep convolutional neural networks solving the problem of single depth map super-resolution. Both networks learn residual decomposition and trained with specific perceptual loss improving sharpness and perceptive quality of the upsampled depth map. Several experiments on various depth super-resolution benchmark datasets show state-of-art performance in terms of RMSE, SSIM, and PSNR metrics while allowing us to process depth super-resolution in real time with over 25-30 frames per second rate.

In this paper, we propose a solution to ease registrational tasks on users and keep them motivated during long lasting procedures using gamification elements. Augmented Reality (AR) technology appears in many different hardware setups ranging from mobile applications to full scale room tracking constructions. Many of which contain non-rigid sensors that require re-registrational maintenance to preserve satisfactory tracking capabilities. While important for a functioning setup, these tasks can become tiring over time leading to less care spent on a qualitative registration. We report on our preliminary study results, showing that a gami- fied registration routine incites participants to perform these tasks for a longer amount of time with up to four times as many measurements taken as the non-gamified version.

Online dense reconstruction is a major task of Augmented Reality (AR) applications; especially for realistic interactions like collisions and occlusions. Monocular cameras are most widely used on AR equipment; however existing monocular dense reconstruction methods have poor performance in practical applications due to the lack of true depth. This paper presents an online monocular dense reconstruction framework using learned depth; which overcomes the inherent difficulties of reconstruction for low-texture regions or pure rotational motions. Firstly; we design a depth prediction network combined with an adaptive loss; so that our network can be extended to train on mixed datasets with various intrinsic parameters. Then we loosely combine depth prediction with monocular SLAM and frame-wise point cloud fusion to build a dense 3D model of the scene. Experiments validate that our depth prediction from a single view reaches a state-of-the-art accuracy on different benchmarks; and the proposed framework can reconstruct smooth; surface-clear and dense models on various scenes with dedicated point cloud fusion scheme. Furthermore; collision and occlusion detection is tested using our dense model in an AR application; which demonstrates that the proposed framework is particularly suitable for AR scenarios. Our code will be publicly available along with our indoor RGB-D dataset.

Camera relocalization is a common problem in several applications such as augmented reality or robot navigation. Especially; augmented reality requires fast; accurate and robust camera localization. However; it is still challenging to have a both real-time and accurate method. In this paper; we present our hybrid method combing machine learning approach and geometric approach for real-time camera relocalization from a single RGB image. We propose a light Convolutional Neural Network (CNN) called xyzNet to efficiently and robustly regress 3D world coordinates of key-points in an image. Then; the geometric information about 2D-3D correspondences allows the removal of ambiguous predictions and the calculation of more accurate camera pose. Moreover; we show favorable results compared to previous machine learning based approaches about the accuracy and the performance of our method on different datasets as well as the capacity to address challenges concerning dynamic scene.

Natural interaction with virtual objects in AR/VR environments makes for a smooth user experience. Gestures are a natural extension from real world to augmented space to achieve these interactions. Finding discriminating spatio-temporal features relevant to gestures and hands in ego-view is the primary challenge for recognising ego centric gestures. In this work we propose a data driven end-to- end deep learning approach to address the problem of ego centric gesture recognition; which combines an ego hand encoder network to find ego hand features; and a recurrent neural network to discern temporally discriminating features. Since deep learning networks are data intensive; we propose a novel data augmentation technique using green screen capture to alleviate the problem of ground truth annotation. In addition we publish a dataset of 10 gestures performed in a natural fashion in front of a green screen for training and the same 10 gestures performed in different natural scenes without green screen for validation. These gesture performances are recorded using the ego centric RGB camera in a HoloLens with corresponding head transformations. We evaluate the performance of our network using our natural dataset. In addition; we also present the results of our network's performance in comparison to the state-of-the-art using the AirGest dataset.

We propose a visualization method for inspecting manufacturing defects. Industrial products have many straight lines and little texture; therefore; the proposed method uses edges for estimating 6DoF pose of the products (registration). To prevent combinatorial explosion; our method reduces the number of combinations by the condition of edges' geometrical distribution. Moreover; manufacturing defects are detected and visualized by robust registration based on the LMedS. This method realizes on-site product inspection for unskilled workers unfamiliar with AR; and decreases the cost of re-manufacturing. We evaluate our method quantitatively using original CG and real image dataset.

In this paper; we propose a user interface which expands the workspace of a smartphone by superimposing a virtual touchscreen on a flat surface such as a desk and the user's lap and allowing the user to perform touch operation on the surface. Information is displayed not only on the flat surface but also outside and above the surface and is fixed in real space; which enables the user to effectively utilize the space and change the area to be displayed quickly by moving the device. The user can drag the entire workspace by swiping the surface or performing gesture operation in the air. This enables the user to move the objects outside the surface or out of the user's reach toward the user so that the user can touch the objects. We implemented a map application to confirm the effectiveness of the proposed interface.

We propose a method that is effective in tracking 3D hand poses occluded by a real object. Since existing model-based tracking methods rely only on observed images to estimate hand joints; tracking generally fails when the hand joints are largely invisible. This problem becomes more prevalent when the tracked hand is grabbing an object; as occlusion by the object makes it harder to find a proper correspondence between the hand model and observation. The proposed method utilizes the occluded part of the hand as additional information for model-based tracking. The occluded depth information is reconstructed according to the geometric of the object and model-based tracking is employed based on particle swarm optimization(PSO). We demonstrate that the reconstructed depth information improves the performance of tracking an object-grabbing hand.

In this paper; we propose a system that can reproduce the material appearance of real objects using mobile augmented reality (AR). Our proposed system allows a user to manipulate a virtual object; whose model is generated from the shape and reflectance of a real object; using the user's own hand. The shape of the real object is reconstructed by integrating depth images of the object; which are captured using an RGB-D camera from different viewpoints. The reflectance of the object is obtained by estimating the parameters of a reflectance model from the reconstructed shape and color images; assuming that the light source is attached to the camera. We measured the shape and reflectance of some real objects and presented the material appearance of the objects using mobile AR. It is confirmed that users were able to obtain the perception of materials from changes in gloss and burnish of the objects by rotating the objects using their own hand.

Unrealistic Mixed Reality (MR) experiences can be caused by unprocessed occlusions between real and augmented objects during interactions. Depth knowledge is indeed key to achieve a seamless visualisation when a bare hand interacts with an augmented object. This can be addressed by blending real-time 3D finger tracking information with the visualisation of the hand in MR. We propose an approach that automatically localises the hand in RGB images and associates the respective depth; estimated with an auxiliary infrared stereo camera used for hand tracking; to each RGB hand pixel. Because misalignments between the outline of the hand and its depth may occur due to tracking errors; we use the distance transform algorithm to densely associate depth values to hand pixels. In this way hand and augmented object depths can be compared; and the object can be rendered accordingly. We evaluate our approach by using an MR setup composed of a smartphone and a Leap Motion mounted on it. We analyse several hand configurations and measure the erroneously classified pixels when occlusions occur.

Occlusion handling is essential to provide a seamless integration of virtual and real objects in AR applications. Different approaches have been presented with a variety of technologies; environment conditions and methods. Among these methods; 3D model-based occlusion approaches have been extensively used. However; these solutions could be too time-consuming in certain situations; since they must render all the occlusion objects even though they are invisible. For this reason; we propose an inverse 3D model-based solution for handling occlusions; designed for those AR applications in which virtual objects are placed inside a real opaque object with holes or windows. With this restriction; the occlusion problem could be solved by rendering the geometry of transparent/hollow objects instead of rendering the opaque geometry. The method has been tested in a real case study with an augmented car in which the virtual content is shown in the interior of the vehicle. Results show that our method outperforms the traditional method; proving that this approach is an efficient option for solving the occlusion problem in certain AR applications.

In wearable augmented reality systems; optical see-through near-eye displays (OST NEDs) based on waveguides are becoming a standard as they are generally preferred over solutions based on semi-reflective curved mirrors. This is mostly due to their ability to ensure reduced image distortion and sufficiently wide eye motion box without the need for bulky optical and electronics components to be placed in front of the user's face and/or onto the user's line of sight. In OST head-mounted displays (HMDs) the user's own view is augmented by optically combining it with the virtual content rendered on a two-dimensional (2D) microdisplay. For achieving a perfect combination of the light field in the real 3D world and the computer-generated 2D graphics projected on the display; an accurate alignment between real and virtual content must be yielded at the level of the NED imaging plane. To this end; we must know the exact position of the user's eyes within the HMD reference system. State-of-the-art methods models the eye-NED system as an off-axis pinhole camera model; and therefore include the contribution of the eyes' positions into the modelling of the intrinsic matrix of the eye-NED. In this paper; we will describe a method for robustly calibrating OST NEDs that explicitly ignore this assumption. To verify the accuracy of our method; we conducted a set of experiments in a setup comprising a commercial waveguide-based OST NED and a camera in place of the user's eye. We tested a set of different camera (or eye) positions within the eye box of the NED. The obtained results demonstrate that the proposed method yields accurate results in terms of real-to-virtual alignment; regardless of the position of the eyes within the eye box of the NEDs (Figure 1). The achieved viewing accuracy was of 1.85 $\pm$ 1.37 pixels.

Most augmented reality (AR) research is performed with commercially-available displays. However; these displays have unadjustable mechanical and optical properties; which limit the experimental questions that can be asked. In order to ask certain questions; it becomes necessary to build a custom display; using off-the-shelf optical components. In the field of visual perception; such devices are often developed; and are called haploscopes. In this paper; we describe the mechanical design of an AR haploscope; which can present virtual objects seen in augmented reality. In order to make accurate measurements; the haploscope must be carefully calibrated; but this calibration is quite difficult. Therefore; this abstract contributes a description of an AR haploscope; and outlines calibration procedures.

Accurate camera localization is an essential part of tracking systems. However; localization results are greatly affected by illumination. Including data collected under various lighting conditions can improve the robustness of the localization algorithm to lighting variation. However; this is very tedious and time consuming. By using synthetic images; it is possible to easily accumulate a large variety of views under varying illumination and weather conditions. Despite continuously improving processing power and rendering algorithms; synthetic images do not perfectly match real images of the same scene; i.e.; there exists a gap between real and synthetic images that also affects the accuracy of camera localization. To reduce the impact of this gap; we introduce "REal-to-Synthetic Transform (REST)." REST is an autoencoder-like network that converts real features to their synthetic counterpart. The converted features can then be matched against the accumulated database for robust camera localization. Our results shows that REST improves matching accuracy by approximately 30\\%.

In Augmented Reality applications; user experience is highly dependent on the accuracy of registration between digital content and the real world. Errors in tracking and registration can arise due to inaccuracy of sensors or challenging conditions such as urban canyon effects or magnetic distortions. Indirect augmented reality is an approach that avoid these issues by using precaptured and preregistered images instead of a live video feed. However; indirect augmented reality highly depends on the availability of those preregistered images. In particular; when being used for browsing geographic information; it is important to access data in an omnipresent way. In this work; we propose an indirect augmented reality browser that aims to address these availability problems by combining indirect AR with crowd sourced precaptured street level imagery with geospatial data. We demonstrate how our indirect augmented reality browser annotates buildings and landmarks in the users' environment and investigate the feasibility by analysing the performance of such an approach. In addition; we investigate issues of visibility and legibility when labelling the environment.

Shared spaces for remote collaboration are nowadays possible by considering a variety of users; devices; immersion systems; interaction capabilities; navigation paradigms; etc. There is a substantial amount of research done in this line; proposing different solutions. However; still a more general solution that considers the heterogeneity of the involved actors/items is lacking. In this paper we present MIME; a mixed-space tri-collaborative system. Differently from other mixed-space systems; MIME considers three different types of users (in different locations) according to the level of immersion in the system; who can interact simultaneously -- what we call a tri-collaboration. For the three types; we provide a solution to navigate; point at objects/locations and make annotations; while users are able to see a virtual representation of the rest of users. Additionally; the total number of users that can simultaneously interact with the system is only restricted by the available hardware; i.e.; various users of the same type can be simultaneously connected to the system. We have conducted a preliminary study at the laboratory level; showing that MIME is a promising tool that can be used in many real cases for different purposes.

In conventional virtual reality systems; users usually do not perceive and recognize the experience as reality. For example; users of a virtual disaster simulator know; consciously or unconsciously; that the presented disaster is not real; which inherently limits the training effect. To make the virtual experience more believable; we propose a novel real-virtual transiton technique that preserves the sense of ``conviction about reality'' in a virtual environment. This is realized by spatio-temporal smooth transition from the real environment to the virtual environment with omnidirectional video captured in advance at the user's position. Our technique requires less preparation cost and presents a more believable experience compared to existing transition techniques using a handmade 3D replica of the real environment. In this article; reported are the concept of our technique; a prototype system and a preliminary user study that has shown the effectiveness of the technique.

This paper describes an Augmented Reality remote collaboration system leveraging high-fidelity; dense scene reconstruction for intuitive and precise remote guidance. A local worker in need of help can use our system to automatically generate a 3D mesh of the scene that is streamed to a remote expert. Thanks to the reconstruction the remote expert can navigate and explore the scene independently of the local worker in full six degrees of freedom. World-stabilized text- and image-annotations can be placed in the scene and strokes drawn on surfaces are placed intelligently in the world. In addition; the reconstruction allows the remote expert to segment objects from the scene and create simple animations with these 3D models to convey precise instructions.

This application paper presents the work of a multidisciplinary group of designing; implementing; and testing an Augmented Reality (AR) surgical telementoring system. The system acquires the surgical field with an overhead camera; the video feed is transmitted to the remote mentor; where it is displayed on a touch-based interaction table; the mentor annotates the video feed; the annotations are sent back to the mentee; where they are displayed into the mentee's field of view using an optical see-through AR head-mounted display (HMD). The annotations are reprojected from the mentor's second-person view of the surgical field to the mentee's first-person view. The mentee sees the annotations with depth perception; and the annotations remain anchored to the surgical field as the mentee moves their head. Average annotation display accuracy is 1.22cm. The system was tested in the context of a user study where surgery residents (n = 20) were asked to perform a lower-leg fasciotomy on cadaver models. Participants who benefited from telementoring using our system received a higher Individual Performance Score; and they reported higher usability and self confidence levels.

The Microsoft HoloLens is one of the latest headsets that facilitates mixed and augmented reality (AR) applications. It has a high potential to leverage AR applications in many domains. However; the first version also comes with several limitations such as the limited battery lifetime and the small field of view. An important one is the lack of high-fidelity depth data which would facilitate object detection and tracking research; a capability imperative for many applications. To mitigate this limitation; we integrated the HoloLens into a point cloud-based tracking system. Our system uses several Kinect range cameras to obtain a point cloud and to detect and track real assets of interest in this point cloud. The pose data for all objects are forwarded to the HoloLens; which can then render 3D models from the right perspective. However; this system is not free of tolerances and tracking errors; which mandates calibrations. This poster explains how the system was set-up and verifies the feasibility of a system such as this for 3D model registration using the HoloLens as a display device.

With data growing at a huge rate; there arises a need for advanced data visualization techniques. Visualizing these data sets in Mixed Reality(MR) mode provides an immersive experience to the user in the context of the real world applications. Most of the existing works can only be used with inordinately priced devices such as Microsoft HoloLens; Meta Glass that use proprietary hardware for data visualization and user interaction through hand gestures. In this paper; we demonstrate a cost-effective solution for data visualization using frugal devices such as Google Cardboard; VR Box etc in MR mode. However; these devices still employ only primitive modes of interaction such as the magnetic trigger; conductive lever and have a limited user-input capability. To interact with visualizations and facilitate rich user experience; we propose the use of intuitive pointing fingertip gestural interface in the user's Field of View(FoV). The proposed pointing hand gesture recognition framework is driven by cascade of state-of-the-art deep learning model - Faster RCNN for localizing the hand followed by a proposed regression CNN for fingertip localization. We conduct both objective and subjective evaluation to demonstrate the performance of our proposed method. Objective metrics are fingertip recognition accuracy and computational time. Subjective evaluation included user comfort and effectiveness of fingertip interaction that is proposed.

We propose to improve virtual reality (VR) and optical see-through augmented reality (AR) head-mounted display scene exploration efficiency by allowing the user to adapt the field of view interactively. This way the user can zoom in to examine parts of the scene in more detail without having to translate the viewpoint forward, as would be required in conventional fixed field of view scene exploration. The user can also zoom out, to gain a more comprehensive view of the scene and to examine distant parts of the scene in parallel, without the need to translate the viewpoint backward. Zooming in is supported with a focus+context visualization approach that integrates a distortion-free magnified focus region seamlessly into context. For AR, the higher resolution focus region is resampled from the video feed acquired by a head-mounted high-resolution camera. We demonstrate the benefits of our free field of view scene exploration in the context of VR and AR tasks, where it brings a substantial reduction of viewpoint translation, view direction rotation, and task completion time.

Camera Time Warp or ``CamWarp'' is an extension of current reprojection techniques for video see-through augmented reality (AR), which significantly reduces the registration error between captured real-world videos and rendered virtual images. Experiment participants were asked to report discomfort while moving their head in a Fitts' Law inspired pattern. Results suggest that CamWarp can reduce discomfort and cybersickness symptoms for all tested camera configurations. In a second experiment participants were asked to move physical objects on a projected path as quickly and precisely as possible. CamWarp had a positive effect on speed and accuracy.

In this paper, we explore how Augmented Reality (AR) and anthropomorphism can be used to assign emotions to common physical objects based on their needs. We developed a novel emotional interaction model among personified physical objects so that they could react to other objects by changing virtual facial expressions. To explore the effect of such an emotional interface, we conducted a user study comparing three types of virtual cues shown on the real objects: (1) information only, (2) emotion only and (3) both information and emotional cues. A significant difference was found in task completion time and the quality of work when adding emotional cues to an informational AR-based guiding system. This implies that adding emotion feedback to informational cues may produce better task results than using informational cues alone.

Over the past few years, playing Augmented Reality (AR) games on smartphones has steadily been gaining in popularity (e.g., Pok\'emon Go). However, playing these games while navigating traffic is highly dangerous and has led to many accidents in the past. In our work, we aim to augment peripheral vision of pedestrians with low-cost glasses to support them in critical traffic encounters. Therefore, we developed a lo-fi prototype with peripheral displays. We technically improved the prototype with the experience of five usability experts. Afterwards, we conducted an experiment on a treadmill to evaluate the effectiveness of collision warnings in our prototype. During the experiment, we compared three different light stimuli (instant, pulsing and moving) with regard to response time, error rate, and subjective feedback. Overall, we could show that all light stimuli were suitable for shifting the users' attention (100\\% correct). However, moving light resulted in significantly faster response times and was subjectively perceived best.

With the increasing acceptance of eye tracking as a viable interaction method for Virtual Reality (VR) headsets, thoughtful gaze interaction methods need to be carefully designed to meet common challenges such as the Midas Touch problem, where users unintentionally select onscreen objects by gazing upon them. This paper presents DualGaze, a novel interaction method in which users perform a distinctive two-step gaze gesture for object selection. Once users gaze upon an object that they wish to select, a confirmation flag pops up next to the object at a location where the users' gaze just passed through. This trajectory-adaptive flag placement strategy reduces the chance of unintentional confirmation by requiring a returning gaze back to the flag. We conducted a user study to compare the accuracy and selection speed of DualGaze and the popular gaze fixation method on a simple gaze-typing task. Our results show that DualGaze is significantly more accurate while maintaining a comparable selection speed that was observed to improve with familiarity of use.

This paper presents a novel evaluation of an industry-ready HMD for delivering AR work instructions in a real-life, industrial procedure for novice users. A user study was performed to examine the potential benefits and limitations of a dynamic 3D virtual model and AR text instructions, delivered through an optical see through HMD, for training users in a new industry procedure (i.e., Yaw Motor Servicing of a wind turbine). Measures of task accuracy and completion time were used to evaluate the performance of one group of mechanical engineering students performing this procedure for the first time guided by AR compared to a second group performing it using a tablet-delivered instruction manual. Results showed AR improved accuracy but not speed of task completion. AR significantly increased accuracy on one specific task-step in the procedure, namely measurement of a thin air gap (see figure 1, left panel), but also showed limitations with other task-steps not benefitting or even being slowed down by AR (see figure 1, right panel). Findings speak to the importance of incorporating an analysis at the level of individual task steps in order to fully evaluate AR work instructions.

The causes and treatment solutions of congenital heart defects are difficult to address and discuss between patient and doctor. This is mainly due to the complex spatial nature of congenital cardiac defects, which makes it difficult for the patients to envision the defect without prior anatomical knowledge and renders the comprehension largely dependent on doctors' (variable) skills to describe the anomaly. To improve communication, 3D printed hearts have been developed, yet these are expensive, difficult to manage for the large collection of defects, and require substantial oral explanation. In addition, the correlation with cardiac function remains rather abstract. Instead, we propose an augmented reality solution, involving a see-through head-mounted display (HMD) extended with a built-in heart rate monitor. In order to increase the presence and the conversational power, the heartbeat of the patient is used to drive an animation of a supersized, floating heart visualisation; enabling the user to inspect a specific heart condition from all sides. To enable this, a universal add-on casing was developed for the HoloLens. Heuristic analysis and pilot tests with 6+15 participants reveal limitations of the implementation and show that the solution does increase comprehension, although more has to be done to enable a robust system.

Human gaze is a crucial element in social interactions and thus an important topic for social Augmented, Mixed, and Virtual Reality (AR,MR,VR) applications. In this paper we systematically compare four different modes of gaze transmission, (1) natural gaze, (2) hybrid gaze, which combines natural gaze transmission with a social gaze model, (3) synthesized gaze, which combines a random gaze transmission with a social gaze model, and (4) purely random gaze. Investigating dyadic avatar-mediated interactions, we found a linear trend for the perception of virtual rapport, trust, and interpersonal attraction, suggesting that these perceptual measures increase with higher naturalness and social adequateness of the transmission mode. We further investigated the perception of realism as well as the resulting gaze behavior of the avatars and the human participants. We discuss these results and their implications.

Text entry is an imperative issue to be addressed in current entry systems for virtual environments (VEs). The entry method using a physical keyboard is still the most dominant choice for an efficient interaction regarding text entry. In this paper, we propose a typing system with a style of mixed reality, which is called HiKeyb, and it possesses a similar high-efficiency with the single physical keyboard in the real environment. The HiKeyb system consists of a depth camera, a pose tracking module, a head-mounted display (HMD), a QWERTY keyboard and a black table mat. This system can guarantee the entry efficiency and the amenity by not only introducing the force feedback from a movable physical keyboard, but also improving the immersion with the real hand image. In addition, the infrared absorption material helps improve the robustness of the system against different lighting environments. Experiments have proved that users wearing HMDs in Virtual Phrases session can achieve an entry rate of 23.1 words per minute and an error rate of 2.76\\%, and the rate ratio of virtual reality to real world is 78\\% when typing phrases. Besides, we find that the proposed system can provide a relatively close entry efficiency to that using a pure physical keyboard in the real environment.

In this paper, we analyze how we perceive the peripersonal space when involved in a reaching task in an Augmented Reality (AR) environment. In particular, we aim to quantify whether distortions in perception of the spatial layout of the scene occur, by taking into consideration two different AR wearable devices, in particular head-mounted displays (HMD). We performed two tests, and compared the results between the subjects who used an Optical See-Through (OST) HMD (Metavision Meta 2) and those who used a Video See-Through (VST) HMD (a smartphone in conjunction with a headset like the Google Cardboard). The data has been collected from a total of 45 volunteer participants. In the experiment, the subjects had to perform a precision reaching task by overlapping the hand on the perceived target position. Then, we observed how the presence or absence of an internal feedback influenced the homing performance. Our results revealed a better depth estimation, thus a more precise interaction, when using the OST device, which also revealed a lower impact on eye strain and fatigue.

We propose a method for quantitatively assessing the quality of Geo AR browsers. Our method aims at measuring the impact of attitude and position estimations on the rendering precision of virtual features. We report on lessons learned by applying our method on various AR use cases with real data. Our measurement technique allows to shedding light on the limits of what can be achieved in Geo AR with current technologies. This also helps in identifying interesting perspectives for the further development of high-quality Geo AR applications.

Digitization and the growing capabilities of data networks enable companies to perform tasks via remote support, which previously required service personnel to travel. But which mixed reality method leads to better results regarding human factors, grounding and performance criteria? This paper reports on a collaborative user study, in which a local worker is guided by a remote expert with the help of different augmented reality methods. The performed task is an controller exchange in a switch cabinet of an industrial robot, a task rather typical for failure detection within the field. Our study was conducted in collaboration with a technician school of which 50 technician apprentices participated in our study. Our results show clear advantages of using augmented reality (AR) to enable remote support. It further gives significant indications for using a projection based AR method.

As head mounted displays (HMDs) become everyday consumer items, the potential of immersive Virtual Reality (VR) as a design space becomes ever more promising. However, their usage is impeded by human factors inherent to the technology itself, such as visually induced motion sickness (VIMS), caused by the disconnect between what is visually and physically perceived. Previous work on VIMS reduction has explored techniques targeting HMDs, while others explored techniques that target the multimedia content itself through visual optimization. The latter are often studied individually and cannot be applied to certain VR content such as 360 video. Consequently, this paper describes an exploratory study comparing and combining such techniques (independent visual background and restricted field of view) in 360 video. The work provides constructive insights for VR designers, while also exploring how analytics of VR content and user experience can be used for VIMS prevention and evaluation.

We propose a framework called inverse augmented reality which describes the scenario that a virtual agent living in the virtual world can observe both virtual objects and real objects. This is different from the traditional augmented reality. The traditional virtual reality; mixed reality and augmented reality are all generated for humans; i.e.; they are human-centered frameworks. On the contrary; the proposed inverse augmented reality (IAR) is a virtual-agent-centered framework; which represents and analyze the reality from a virtual agent's perspective. In this paper; we elaborate the framework of inverse augmented reality to argue the equivalence of the virtual world and the physical world regarding the whole physical structure.

We designed a within-subject experiment to compare visual discomfort to preteen users caused by using head-mounted displays (HMD) and tablet computers for an hour. 18 participants younger than 13 years old were recruited to fulfill a series of similar painting tasks under both display conditions. Visual fatigue was measured with visual analog scale before and after experiment and during the break of experiment. The results indicated that HMD had a trend to bring higher visual fatigue than tablet computer during the exposure of 1 hour. Although the symptoms of visual discomfort disappeared after resting; there is need for preteen-specific head-mounted displays.

Both digital and physical 3D models of buildings as well as historical photographs of architecture are used for a wide range of needs; from research in humanities and information technologies; museum contexts and library studies; to touristic applications. Spatially oriented photographs play an important role in visualizing and browsing contemporary as well as historical architecture; starting with the ground-breaking Photo Tourism project [6]. We present a technique to combine physical 3D models of buildings with spatially registered historical photographic documents in a hand-held Augmented Reality (AR) environment. Users are enabled to spatially explore historical views of architecture by selecting photos from a collection of images which are then utilized as textures for the physical model rendered on their respective mobile device. We compare different methods to select photos registered to a physical model in hand-held AR.

The Mixed Reality (MR) technology can be used to create unique collaborative experiences. In this paper; we propose a new remote collaboration platform using MR and eye-tracking that enables a remote helper to assist a local worker in an assembly task. We present results from research exploring the effect of sharing virtual gaze and annotations cues in a MR-based projector interface for remote collaboration. The key advantage compared to other remote collaborative MR interfaces is that it projects the remote expert's eye gaze into the real worksite to improve co-presence. The prototype system was evaluated with a pilot study comparing two conditions: POINTER and ET (eye-tracker cues). We observed that the task completion performance was better in the ET condition. And that sharing gaze significantly improved the awareness of each other's focus and co-presence.

We present a diminished reality application running live on consumer mobile devices. In our pre-observation-based approach; the clean 3D scene; free of undesired objects; is scanned beforehand and reconstructed as a high resolution textured 3D model. At runtime; objects added in a region of interest are efficiently removed by projecting the previously captured background. Differences of illumination conditions between scan time and run-time are compensated to obtain seamless results. The proposed approach requires no segmentation or manual input other than the definition of the 3D region of interest to be diminished; and is not based on any particular assumption on the background geometry. We show the potential of our approach by processing a variety of challenging unknown 3D scenes including textured backgrounds; dynamic illumination conditions and foreground objects partially occluding the diminished region. We provide details on our compute shader implementation to make as easy as possible the reimplementation by the community.

Supporting maintenance workers with Augmented Reality (AR) applications has always been one key use case to demonstrate the advantages of AR. However; such AR applications are still not widely used in industry; which may lay at the complex industrial requirements. We present the user; technical; environmental and regulative requirements for an AR maintenance worker support system; which were gathered by analyzing three divers production sites.

The transition from high school to university is an exciting time for students including many new challenges. Particularly in the field of science; technology; engineering; and mathematics; the university dropout rate may reach up to 40\\%. The studies of physics rely on many abstract concepts and quantities that are not directly visible like energy or heat. We developed a mixed reality application for education; that augments the thermal conduction of metal by overlaying a representation of temperature as false-color visualization directly onto the object. This real-time augmentation avoids attention split and overcomes the perception gap by amplifying the human eye. Augmented and Virtual Reality environments allow students to perform experiments that were impossible to conduct for security or financial reasons. With the application; we try to foster a deeper understanding of the learning material and higher engagement during the studies.

Many techniques have been proposed for navigation using head-mounted-displays (HMDs) in virtual reality (VR). A walking-in-place (WIP) interface for virtual locomotion provides a high presence and an immersive experience in a virtual environment (VE). However; most of the WIP techniques can only navigate users in the direction of their gaze. Other methods considering virtual locomotion direction have complex configurations or are only feasible when the tracking spaces are not limited. This paper proposes a WIP interface independent of gaze direction based on the data analysis of waist-mounted inertial sensors. Our method can navigate in the locomotion direction by calculating the orientation of the pelvis. We experimentally compared two WIP methods using a navigation task that required participants to periodically observe the surrounding VE: (1) Conventional WIP (gaze-based direction) (2) Proposed WIP (pelvis-based direction). While there was no difference in learnability or cybersickness between the two methods; the proposed method had shorter task time and higher efficiency.

In this poster; we explore social sharing of surrounding environments on wearable Augmented Reality (AR) devices. In particular we propose filtering the level of details of sharing the surround environment based on the social proximity between the viewer and the sharer. We test the effect of having the filter (varying level of details) on the shared surrounding environment on the sense of privacy for both the viewer and the sharer perspectives. We conducted a pilot study using HoloLens. We report on semi-structured questionnaire results and suggest future directions in the social sharing of surrounding environments.

``Cybertravel`` in virtual reality (VR) system can easily provoke cybersickness as there are no vestibular cues available during visual optic flow. This study examined the effect of navigation speed as well as the use of different VR devices for navigation on cybersickness. Participants experience street navigation while they are standing still. Four conditions: CAVE (cave automatic virtual environment) and HMD (head-mounted display) x 10 m/s and 24 m/s; were tested while participants perform a counting task. Results showed that higher navigation speed leads to increases in ratings of severity of cybersickness measured by simulator sickness questionnaire (SSQ) and miserable score (MISC). A difference in cybersickness ratings between VR devices is also observed with experimental order effect.

Voice-activated Intelligent Virtual Assistants (IVAs) such as Amazon Alexa offer a natural and realistic form of interaction that pursues the level of social interaction among real humans. The user experience with such technologies depends to a large degree on the perceived trust in and reliability of the IVA. In this poster; we explore the effects of a three-dimensional embodied representation of Amazon Alexa in Augmented Reality (AR) on the user's perceived trust in her being able to control Internet of Things (IoT) devices in a smart home environment. We present a preliminary study and discuss the potential of positive effects in perceived trust due to the embodied representation compared to a voice-only condition.

One of the challenges in reaching wide-spread autonomous driving is the establishment of driver trust in the technology. We suggest a windshield display interface showing the perceptive abilities and decision-making of an automated car while driving. We took a human-centered design approach to determine user expectations and requirements. We present our resulting interface prototype which runs in a mixed reality environment. We plan to evaluate its impact on situation awareness and trust in hard-to-predict urban scenarios.

A proposal was first made in 1971 for a study attempting to investigate radical constructivism as a valid learning theory; though the study was never formally conducted. This work describes our Virtual Reality interactive four-dimensional Hypercube system used as our investigative medium; and our initial implementation of the historic study proposal for validation. Our lessons learned are leading to further experimentation and investigation into learning applications in Virtual Reality.

Virtual reality has been used intensively to study embodiment and body perception; in particular for research purposes in psychological domains. Virtual avatars are used to resemble users' appearance and to implement interactively simulated behaviour. An important requirement to experience the bodies in the virtual environment as realistic; is that users should feel embodiment; i.e. Ownership; Agency; Self-location/Presence. State-of-the-art capture and display technologies allow for extending virtual reality embodiment to the realm of augmented reality for higher efficacy---instead of seeing a virtual reality body one would see a captured; 3D representation of their own body naturally controlled by their real body movements within the context of the present real environment. However; it is unclear whether users would experience embodiment with their augmented reality avatar and whether findings from virtual reality targeting body perception can be replicated. Here we present an augmented reality system comprising a 3D point cloud capturing system (Microsoft Kinect) and an optical see-through head-mounted display (Microsoft HoloLens); both connected to a purpose-developed application displaying a user's body in a virtual 3D mirror embedded into the real environment. In a study with 24 participants; we evaluated embodiment and replicated parts of a virtual reality study on body weight perception as a proof of concept. Our findings show that users experience ownership and agency with the mirrored body and that body weight perception in virtual and augmented reality systems is similar.

According to the advance of tracking technology; wireless technology; battery capacity and social acceptance for Head Mounted Display (HMD); the user has been able to not only overlook the virtual space while sitting on a chair; also play Virtual Reality (VR) more dynamically while walking around a room or maybe at outdoor. On the other hand; there is possibility that the user collides with surrounding obstacles and gets hurt because of walking around in real space without removing the HMD. In this paper; in order to support the user for reaching a destination in real space in safety while playing VR game; we propose the walking support system that display a route to the destination in virtual space using Social Force Model that is a model simulating how pedestrians move in the presence of obstacles and other pedestrians.

Museums and exhibitions often involve physical artefacts which may contain rich histories or deep meaning associated with them. These additional contents are often installed physically as informational panels shown on a wall display. However; it may sometimes be challenging to deploy due to space constraints. In order to address this challenge; we introduced the use of mixed reality. Mixed reality offers an immersive and interactive experience through the use of head mounted displays and in-air gestures. Visitors can discover additional content virtually; without changing the physical space. For a small-scale exhibition at a cafe; we developed a Microsoft HoloLens application to create an interactive experience on top of a collection of historic physical items. Through public experiences at the caf\'e; we received positive feedback of our system. In this paper; we discuss the design and implications of our system; survey results; as well as challenges that were encountered in deploying our mixed reality experience in a public setting.

Virtual Reality has the potential to transform the way we work; rest and play. This promise comes with new challenges. One challenge stems from the interactive nature of immersive Virtual Environments (VEs). Placement of contextual information in VEs can be critical to the user experience. This poster describes the use of eye-tracking to alleviate usability issues surrounding information presentation in immersive VEs. Results from our experiments show that integrating eye tracking into a VE to dictate where and when textual information is presented can improve performance when searching for contextual information. In summary; the results show improvement in task performance when the new direct method is employed to reveal information in target regions based on gaze. This seems to hold true independent of the VE or the type of information questioned.

We present a concept of emotion sharing and augmentation for collaborative mixed-reality. To depict the ideal use case of such system; we give two example scenarios. We describe our prototype system for capturing and augmenting emotion through facial expression; eye-gaze; voice; physiological data and share them through their virtual representation; and discuss on future research directions with potential applications.