News

Ever flipped between some before and after photographs of a place or situation? It can be fascinating, triggering leaps of imagination to fill in the gaps between the two time periods. Now, update this analogue process to 3D VR, AR, and online information. Engaging with history has been rebooted, and it is called situated simulation.

Standing amidst iconic ruins, you are there, but only physically. Absorbed in a history book or engrossed in a film, you can be transported there, although rarely, if ever, physically. The digital devices in our pockets seemingly offer a way to bridge this gap between place and a deeper understanding of it, though Wikipedia, or similar, also come with limitations. A belief that visiting sites of historical significance could be a much richer and more rewarding experience has motivated a Norwegian-led international research collaboration to spend the last 15 years carving out a new genre of digital media—situated simulation, also known as Sitsim.
The core idea behind Sitsim is to create an alternate virtual version of the physical spaces on a mobile digital device. These device screens can portray a wide variety of "simulated" scenarios, such as 3D reconstructions of the built and natural environment, old-new "re-photography", integrated documentary videos, and onscreen links to detailed information, all "situated" within a physical space. Moreover, technology such as GPS, accelerometers, and gyroscopes allow the screen's representation to match the user's location and direction, presenting a virtual overlay narrative on top of the physical space before their eyes.
Essentially, Sitsim creates a virtual window into another time, presenting users with a rich audio-visual experience of being immersed in a simulated version of a location's history, while physically located in its present reality. Although it contains both virtual and real elements, the overlay isn't compressed into a single augmented reality screen and passively observed, instead, it relies on reconciling the dual perceptions inside a user's mind. What takes place is a journey for the imagination, offering a richer engagement with culture and heritage and abundant possibilities for educators and information designers.

A different perspective on AR
The origins of Sitsim lay in a 2007 Norwegian Research Council project into communication genre development and design. Gunnar Liestøl, who heads up Sitsim, was one of the professors at Oslo University exploring new genres made possible by technology, including location-based sensors. While others were interested in virtual overlays common in augmented reality, Liestøl had the idea of using portable screens to create a virtual version of the surrounding space and letting the viewer take in the difference in perspectives inside their head. He explains, "It was not quite augmented reality, so we called it situated simulations, or Sitsim, and is now becoming known as indirect augmented reality, a mix between the screen and the real reality before someone's eyes".
This double perspective was fascinating for Liestøl, reminding him of Gregory Bateson's idea of "double descriptions", whereby two descriptions that overlap tend to generate a different type of information that is neither one of the descriptions. "In a similar way, the perspective from Sitsim you see on the screen and in reality is an oscillation of perspectives that allows new possibilities in an information design sense. Matching perspectives is the overarching design principle of the Sitsim project, where we explore how perspectives can match and how can they interplay and interact in a way that can't be reduced to any one of the perspectives."
Liestøl continues, "Looking at a movie or representation of a place at home is less than half the experience of being there, but being there and removed from background and depth of narrative is also a diminished experience. It is a fascinating and special experience to have this double experience that deepens engagement." He points out that such a perspective contrast is not entirely new, as "re-photography" is an established mode of comparing old photos in contemporary context, but this is static, whereas Sitsim is dynamic and can move through space and easily change perspective.

Trailblazing augmented reality storytelling
In the early stages of the project, the focus was on directly matching the two perspectives, but as the scope of work grew, so did the challenges and technical possibilities. Liestøl brings up the example of Sitsim in the Parthenon. "We discovered that we could move the camera away from the perspective of the real world user. For instance, the Parthenon frieze is 12 metres up in the air and removed from the sight of visitors, so we elevated the virtual camera to that height so the user could actually see it and then walk around and observe it. These different usages of the virtual camera allowed us to generate perspectives that were incongruent with the real perspective, but at the same time maintained the double perspective".
Another compelling example he gives is the Temple of Caesar in Rome, a space people are unable to enter, "Virtual cameras made it possible to actually go and look inside when you are there. This is a really useful feature in cultural heritage because often you are not allowed to walk inside the physical ruin, or on top of it. Yet, you can still get the feeling of being able to do this as part of the experience of being in a place".
Accessing challenging spaces is one advantage, but the advance of 3D technology and photogrammetry scanning has also deepened storytelling possibilities, allowing the retelling of stories and scenes in history. For example, the D-Day landing of US forces on Omaha Beach is reenacted virtually as your smart device scans the former battlefield. Sitsim is adapting its perspectives to easily jump in time and space, and there are now numerous place-based virtual re-enactments, ranging from Viking trading villages to ancient sea voyages inspired by shipwrecks at the bottom of the Aegean Sea.

The metonymy of media design
The ancient Crete port of Phalasarna is instructive in the possibilities for Sitsim, and its ability to recreate history from remaining fragments. Something Liestøl equates to the concept of metonymy, where a part can be used to represent the whole. "Archeology is a metonymic experience where the fragments represent the past, and they are taken up and used as narrative hooks and generators to represent something that happened long before. We did this with the catapult stones found in Phalasarna harbour, where our 3D re-enactment focused on a time when Rome was trying to crush piracy, creating a narrative around an assault on the fortified Phalasarna harbour frequented by pirates".
Same location, different era, a Sitsim tour of Phalasarna also includes the history of a sunken ship from the Battle of Crete in World War 2. Not only does the Sitsim guided tour includes a stereo-phonic 3D recreation of German aircraft sinking the ship, but users can also virtually "dry dive" in the waters around the wreck. Such mixing of location-based context and historical context works so well because Sitsim is able to overcome the limitations of reality. "Essentially, the Sitsim project is about finding historical places and using the space to tell the stories in new ways," says Liestøl.
Sitsim is pushing the boundaries of media and information design, mapping out the new realities offered by VR and AR, a process Liestøl sees as firmly grounded in narrative theory. "Theory is a really important topos for Sitsim storytelling. Such theories are drawn from analysis of literature and film, etc., but now we are trying to use the same concepts the other way around for synthesis. Effectively using these theoretical concepts as rules for the construction of narrative rather than just analytical practice". Narrative potential can be enhanced further by including elements such as gamification, as in the case of Narva. Here the Estonian city's architecture, which was largely destroyed by war, has been recreated by overlaying old photos onto the present cityscape. Once a series of perspective-matching tasks have been completed, the Narva cityscape is re-animated to a full-colour 3D representation.
Another approach to re-animating perspectives lost to time can be found in the Norwegian fishing village of Vágar where photos from the late 19th century became the raw material for recreating 3D animated everyday life stories of people featured in the photos. 3D animation, set in realistically re-created settings, is poised to become a more prominent feature in Sitsim. With plans underway for recreating iconic scenes such as the killing of Julius Caesar using motion capture technology, Liestøl is excited about the prospects for achieving an increased level of authenticity and less mechanical movement for future versions of Sitsim.
For those with an eye to using the technology, Sitism AR editing platforms and plug-ins for the Unity Engine to create and seamlessly integrate 3D animations are making the platform easier to use. Liestøl says they want to open up Sitsim to anyone who wants to experiment with it, and it is only getting easier to use and export to different operating systems, including, in the future, smart glasses and various forms of 3D VR devices.