“We’re never going to be totally immersive as long as we’re looking at a square, whether it’s a movie screen or whether it’s a computer screen.… We’ve got to put the player inside the experience, where no matter where you look you’re surrounded by a three-dimensional experience. That’s the future.” —Steven Spielberg
Imagine stepping inside a movie, being able to turn your head and seeing the scenes all around you as if you were there. Imagine this cinematic experience reacting to you in a way that makes it feel as if the unfolding scenes are influenced by your presence. A door opens when you turn towards it. A detail comes into focus in full stereoscopic view when you turn your attention to it. When your eyes shift from a nearby object, the focus shifts so naturally that you don’t even notice.
This kind of truly immersive cinema that goes beyond the boundary of the static frame and gives way to a truly new kind of experience has been a longtime dream. Aldous Huxley described something like it in his 1931 novel Brave New World. He called it “the feelies.” Our modern pop-culture equivalent is Star Trek’s “holodeck”; a medium that can take us out of our reality and take us somewhere else completely. Both represent our deep desire to merge with images and step through the frame, like Mia Farrow in Woody Allen’s 1985 The Purple Rose of Cairo, and be in the world conjured up on the screen.
Fast-forward to 2013. I am on a plane with a documentary crew headed for Kangerlussuaq airport situated in central-western Greenland. It is our first stop in a 30-day journey that will continue on by ship along the western coast of Greenland across the ice-filled expanse of Baffin Bay to Pond Inlet, an Inuit community at the north end of Baffin Island in the Canadian Arctic.
In our luggage we are carrying 40 GoPro cameras. Our mission: to create the world’s first narrative 360-degree video documentary. It’s a big experiment and a complete step into the unknown for all of us. As the plane makes its final approach, over the breathtaking and hostile-looking 1.7 million square-kilometre ice expanse that covers 80 per cent of Greenland, we pass over the Russell Glacier, where we will go the day after tomorrow to launch our drones; their payload is a 900-gram spherical camera that will allow us, for the first time ever, to shoot airborne 360-degree HD video panoramas in the Arctic.
There has been a quiet revolution going on in imaging technology that is taking monumental steps towards a completely different form of cinema, which will fundamentally change our relationship to the screen and cinema-based arts. There is an astounding amount of research going on worldwide at companies and research institutions from Paris to Silicon Valley working on imaging solutions in areas where even the terms are unfamiliar to most filmmakers. Who has ever heard of video stitching, spherical imaging, 360-degree camera arrays and Virtual Reality Cinema? These terms define a new frontier in imaging technology that is the result of the emergence of fresh ideas, new technologies and the ever increasing miniaturization and computational power of everyday computers, smart phones and cameras.
At the heart of immersive cinema is the ability to film the world by capturing a full sphere. The resulting 360-degree video is just like conventional film that plays in full motion and audio with one revolutionary difference: Because 360-degree camera systems capture a full sphere, the screen has no boundaries and the audience can shift their perspective in any direction in real time while viewing the film.
As a filmmaker and interactive creator I was astounded the first time I saw an example of 360-degree video in early 2012. Anyone interacting with 360-degree video footage immediately grasps the enormous creative promise and potential of seeing reality in this completely new way.
Not long after that initial viewing, my company DEEP was approached by ARTE CEO Wolfgang Bergmann to conceive of an interactive component to complement a 10-part TV series that explores the mythical North West Passage and the profound changes that climate change has brought to the Arctic. It seemed like the ideal project to undertake a groundbreaking experiment in immersive filmmaking. Rather than just showing the Arctic on traditional film, we could use 360-degree video to do something that TV can’t do: we could literally take our viewers to the Arctic and let them look around for themselves.
Our preliminary research was sobering. There were a handful of companies like Yellow Bird that provided solutions and services for the capture of 360-degree video. They charged $15,000 a day for production services and exorbitant sums for their proprietary camera systems. As a consequence, the first pioneering examples of 360-degree video were used as highly effective marketing gimmicks to sell products, applied to scenes of spectacle in well-worn genres such as extreme sporting, supported by wealthy sponsors such as Red Bull. There were no examples of an innovative narrative use of this new medium. It was quite clear that the high cost of 360-degree production had kept the necessary tools out of the hands of independent filmmakers and interactive creators, stifling the creative and narrative evolution of the medium for years.
Fortunately this was about to change with the emergence of two revolutionary new technologies, 3-D printing and the ubiquitously available GoPro camera. A number of pioneers, such as Michael Kintner of 360 Heros and Joergen Geerds of NY-based Freedom 360, created 3-D printed rigs that hold multiple GoPro cameras in spherical arrays, allowing relatively low-cost production of HD spherical video for the first time. Because of their light weight and the fact that they do not need to be tethered to a computer, which made their predecessors so cumbersome, they can be mounted on drones and other remote-controlled devices, greatly expanding their cinematic potential in the field.
When we arrived at the foot of the Kangerlussuaq Glacier in August 2013 to shoot our 360-degree flyover, we had just received the first 3-D printed GoPro holders. Up until then we had strapped our GoPros to broom poles with rubber bands to shoot our spherical test footage. A week before our Arctic expedition, GoPro released a firmware update that kept the cameras from crashing regularly, something that had filled us with fear and dread during our testing phase. Individual cameras would randomly shut down in the middle of a take and when you are shooting in 360 degrees, if any one of the six cameras fails, the resulting shot is unusable.
Had our shoot been a week earlier the results would have been disastrous. This is what it means to be riding the bleeding edge. I have never undertaken a project with so many unknowns stacked on top of each other.
As the drone pilot and I stood facing the glacier, with its shimmering blue and white walls that protrude 75 metres into the air, I wondered whether utilising this technology for the first time in the remoteness of the Arctic was such a great idea.
Our Greenlandic guide indicates an imaginary line that we should stay behind situated a hundred metres from the glacier. The 20,000-year-old ice cracks without warning with a thunderous sound that sends apartment-building-sized boulders and debris rolling ahead of the glacier for hundreds of metres. If our drone were to come down too close to the glacier, it would be unwise to retrieve it.
One of the biggest challenge in shooting 360-degree video is that the camera sees everything in all directions, which means that the crew has to find some place to hide. The terrain at the base of the glacier is so completely flat that the only way for us to do so is to wear camouflage suits that blend into the landscape as the drone flies over the wall of ice.
This establishes a familiar shooting pattern. Roll camera…HIDE! It is odd to shoot scenes while hiding and there are presently no remote monitoring systems. Even worse, when you return after a long day of shooting you can’t view your rushes because it takes days to render the 360-degree footage through an elaborate process. You have to work on faith and it makes you feel a real kinship with the early pioneers of cinema, who were also essentially shooting completely blind.
In 1885 the Lumière Brothers captured footage of a group of women leaving a factory entrance. This 46-second fragment is considered the first piece of film ever shot. It exists because of an earlier revolution in imaging technology: the movie camera. Like 360-degree video, it was a technical innovation that allowed the photographic capture of a moment in full motion. At the moment of its birth, cinema was not yet an art—it was a technical novelty. The language of film, including cuts, close-ups and parallel action, had to evolve over decades through the incessant experimentation of filmmakers trying to tell a story.
360-degree video is in the same place. It is an aesthetic possibility that has come about through the simultaneous emergence of a number of key technologies. The visual language conventions to tell a story in 360 degrees still needs to be discovered and evolved. There are currently no narrative examples that attempt to tell a human story with depth and complexity. Our spherical Arctic documentary is a first humble step in that direction.
What we did not realize at the time was that our journey to create immersive narrative experiences utilising 360-degree video was about to take an unforeseen and very exciting turn with the arrival of a package at our Toronto office in November 2013.
Inside was the Oculus Rift, the first prototype of affordable virtualreality goggles. The device streams stereoscopic 360-degree images that shift in sync with the head movement of the viewer, making them feel as if they are physically in the environment they are seeing.
The Oculus Rift was an innovation driven by gamers who dreamt of being immersed in the elaborate computer-generated worlds of first-person shooters. Using the Rift to display real-life cinematic scenes was not yet part of the agenda and so we were curious to see what 360-degree videos would look and feel like in virtual reality. There was no immediate way to find out. We first had to build a viewer using a gaming engine to play back our footage in the Rift. The first time we immersed ourselves in the footage we had shot at the Kangerlussuaq Glacier was an unforgettable moment. It felt like no other experience we had ever had; it felt like “being there.” Standing on the carpet in our office we were looking up at walls of ice as if we were standing there right in front of them. When we took our VR goggles off we were left with a distinct feeling that we had just been somewhere else.
At first it was hard to describe what made this experience so special, but then it sunk in: Something momentous had occurred. In our little office in Toronto, we had stepped across a threshold that no previous media could cross. We had broken the fourth wall.
Like Mia Farrow in The Purple Rose of Cairo, we had stepped into the movie. As adventurers, we had passed through the separating barrier between art and life, a threshold that has fascinated and challenged artists since the beginning of time.
One of the first modern efforts to consciously break down the fourth wall was undertaken by Richard Wagner, who believed in what he called Gesamtkunstwerk, a concept that foresaw the fusion of all arts into one powerful medium that encompassed everything: sound, voice, light, image, mind and body sensation.
To realise his dream Wagner had a festival house constructed in Bayreuth, Germany, in 1876 where he applied new theatrical innovations— surround-sound reverberance, the isolation of the individual through total darkening of the house and the revitalization of the Greek amphitheatrical seating to focus audience attention—to create, if one is to judge by the accounts of the day, near delirious immersion.
Wagner’s effort to abolish the borderline between object and observer, stage and audience, artwork and spectator 10 years before the advent of cinema, foreshadowed the experience of virtual reality by 140 years.
Back in our Toronto office, we exposed nearly 100 people of all ages and backgrounds to our VR experience and interviewed them right afterward to gauge their reactions. Their fervent enthusiasm took us by surprise. We did not come across a single subject who was not touched or amazed by the experience.
We gained some vital insights into this medium by talking to our test subjects. Nearly all reported that it felt odd that they could not see their own body. At first we concentrated on this statement of the missing body as a shortcoming to address in future experiments and research. Then we realized that this observation was expressing something much more profound and important.
Subconsciously our test subjects felt that the experience was so real that their mind “expected” their bodies to be “there.” This hints at the fact that cinematic VR experiences affect people on a much deeper level than merely watching a film. Their sensory inputs, their ears and eyes, are not reading the input of the VR cinema experience as a codified language that “describes” a moment as traditional film does, but as direct sensory input of a place and time that tricks the brain into feeling that it is “there.” Watching a cinematic VR experience seemingly stimulates the perceptual systems of the brain making the immersion into the content an experiential instead of an intellectual process. Our test subjects described this state as a “sense of presence.” One 70-year-old subject told us that he felt like he was “pure spirit.”
This statement reminds me of one of the core ideas of romanticism, the transcendence of the confines of the body through the spirit. Wagner had a real distaste for the corporeal, the gross vulgarity and fallibility of the body, and the Romantic zeitgeist revolved around the immersion and dissolution of the human body into the unfathomable mystery of nature that surrounds us; Wagner’s Gesamtkunstwerk sought to take this to an extreme by creating the perfect communion with art by controlling all the senses of the body, including sight, sound, time and space. It was something that was never going to be possible in his time.
This feeling of “presence” cannot be created by any previous medium and is unique to VR. It is the great unifying medium of the future that Wagner was trying to find, his Gesamtkunstwerk brought back to life by the magic of modern technology.
All previous media is able to create immersion, through symbolicmagical thinking, wherein the codified narrative of a book, a verbal storyteller, the theatre or film, evokes a place and time and moment in the audience’s brain through description. To transcend it, audiences temporarily suspend their disbelief, taking the visual and aural cues to create a reality that lives solely in their imagination, borne out of descriptive metaphor.
When we gaze on a painting, it is a pictorial representation of a moment. It is not the moment itself. When we read the description of a moment in a book, it isn’t the moment itself. When we watch a film, its highly codified visual language allows us to suspend our belief, so that we can take the description of the moment for the moment itself.
In VR cinema this principle is turned on its head. In VR, images enter the brain through sensory input in the same way that we perceive reality and so it is astounding to see that even at low resolution, that sense of experience and presence is immediately instilled because it instantly triggers deep-rooted sensory and cognitive mechanisms by which we perceive the world not through descriptive information but directly through our senses.
A clear indication of how strongly the subconscious brain perceives our demo as reality is the fact that more than half our test subjects waved involuntarily and said ‘hello’ when they were in a scene where they discovered a person standing behind them. They were also clearly affected when they were put in a scene that was shot from the pontoons of a helicopter flying over icebergs in the Arctic. No matter how much they told themselves that this was not real and that they were standing on a carpet in our office, their subconscious brain overruled their intellectual assessment. They were trying to suspend their belief in a stark reversal of how traditional media works.
Hollywood is taking notice. Jaunt VR, a company whose sole focus is developing hardware for the creation of VR-based cinema, raised $6.8 million in VC financing in April 2014. Jaunt’s board of advisers is recruited from the ranks of Netflix, Dolby, Imax and RED.
Spike Jonze recently announced that he plans to make a Rift-based film.
What this indicates more than anything is that immersive VR cinema is being taken seriously by prominent American production executives and that we are witnessing a phase change in media.
On March 25, 2014, a month before the public launch of Jaunt, Facebook bought Oculus for $2 billion. The acquisition will fund the rapid further development of VR technology at a critical time and will subsidize the final product to keep it affordable when it hits the consumer market in 2015.
Sony and Samsung are expected to launch consumer VR products in the same time frame. Apple and Microsoft have filed for a number of patents for VR headsets and are quickly making up for lost time.
More importantly, since Facebook sees the Oculus Rift and VR as a platform akin to mobile and plans to bring their 1.3 billion users into this experience, it has catalyzed the industry into understanding that what we are witnessing is the birth of a new mass medium.
So VR-based cinema, which is a subset of all the possible uses of virtual reality, will likely have a mass audience in the not-so-distant future. But what will that experience be like? How will cinematic stories be told in VR that utilise its unique attribute of presence?
The power of film lies in the juxtaposition of a dream-like stream of images into a narrative where time and space are condensed through the magic of editing. In film the moment is not a true moment but a metaphor of a moment. The audience connects the rapidly edited fragments of disconnected moments into a coherent reality through their capacity for symbolic-magical thinking.
VR breaks the fourth wall and communicates directly with our senses. Time is time and space is space, each of these dimensions necessitating a cognitive continuity around the way we perceive reality.
It could very well be that the importance of upholding a coherent sense of reality in VR that an audience can relate to will end up being the fundamental organizing principle that will shape the narrative aesthetics of VR.
Film requires the suspension of disbelief to accept the reality conjured up on the screen. In VR the intellect does not even have the capacity to overrule our hijacked senses from which our brain constructs reality because this process happens before we can formulate a thought about what we are seeing. Being takes precedent over thinking.
When we stand at the edge of a precipice in a VR experience, our mind and body react as if we are standing at the edge of a precipice in real life. The reaction to sensory input is primal and non-intellectual.
Using the fragmentary language of film could very well feel schizophrenic precisely because it ignores the dimension of real time and space, dimensions that are at the core of “presence” in VR.
The key to finding a narrative aesthetic for VR most likely lies in the realm of dreams. Dreams, while not being reality, have a sense of presence that we accept as reality while they occur and yet they are laden with emotions, meaning and a narrative beyond our control.
In an unforgettable scene from Cocteau’s first film, Blood of a Poet, he passes through the threshold between art and life, reality and dream by way of a mirror.
Utilising the Oculus Rift to step into another world reminded me of Cocteau’s mirror journeys into a parallel universe.
Filmmakers like Cocteau, Murnau and Bunuel were quite conscious of the fact that film has a trance-like quality and owes a great debt to the narrative structure of our dreams. It is the artform closest to language of dreams, most likely because it is mapped to how our narrative brain works.
Modern cognitive psychologists believe that our perception of reality is like a dream imposed upon the world and that our way of perception and narrative processing of reality is dreamlike.
When we dream of a precipice we have the same reaction as standing on a precipice. Dream, life, virtual reality: There is a powerful connection between the three.
My hunch is that VR cinema will have to find an aesthetic that upholds a sense of presence that connects the viewers gaze to memories and associations in a dream authored by the creator of the experience. In this way filmmakers and interactive creators working with VR in its first hour will have to use trial and error to find the necessary narrative conventions that work, just like the early pioneers of film did.
The final form is likely to feel familiar to us, but unlike anything that we have ever seen before.
Watch The Polar Sea at TVO and explore the online interactive doc!
