We noticed Ethan Shaftel's work some time ago, his virtual reality films with fantastic, funny and terribly enthusiastic universes. Coming from the world of animation, Ethan never stops playing with the viewer to better understand the mechanisms and the use of virtual storytelling. A good opportunity to meet him while he is selected this year at Tribeca Immersive and XR@Annecy with his real-time interactive project AJAX ALL POWERFUL and at Cannes XR with his Annie-nominated animated 360 video KAIJU CONFIDENTIAL.
Coming to immersive from an narrative film background
Ethan Shaftel - My background is largely in traditional film. I was an editor for a long time and I made short films and independent features years ago. But I've always been highly interested in interactivity. And some of my early experiments and even student films were interactive. You watched it on a desktop computer, with some amount of clicking or branching or other techniques that I was still trying to understand. I've never been much of a gamer, which I think of as requiring a “lean forward” posture for the player, as much as I enjoy cinema, which is instead a “lean back” posture for the viewer. That “leaning back” is abandoning yourself to the narrative. So my experiments were all about how to use interactivity, but remain in the “lean back” posture of abandoning yourself, instead of anything that requires puzzling something out, or even being aware of making decisions.
E. S. - I was doing this before there was really an outlet for work like that, before there was a New Frontiers program at Sundance or anything similar that I was aware of. People would ask the question, “is it a game or a movie?” And I didn't have a good answer for that, because it was neither, and for different reasons. So those experiments and prototypes didn’t do anything for me professionally and I moved on to different things
E. S. - When this wave of VR became something that people were talking about, because of Oculus, four or five years ago, I realized that this might be the medium that I have been looking for. It is fundamentally interactive in a way that cinema is not, because of the concrete point-of-view “inside” VR and the built-in agency that requires the visitor to look around The question of “is it a game or a movie” just doesn’t come up, because the answer is that it’s a VR experience and you leave it at that. I began making prototypes and VR storyboards and even remaking my earlier desktop experiments in VR , using tools already becoming available in animation software.
E. S. - The pre-visualizations that we were making to try to get funding for live-action VR projects were actually very close to the quality of some of the finished pieces I was seeing at festivals. So that basically led to making EXTRAVAGANZA, and animated piece and my first VR project, which was presented at Tribeca Immersive in 2017. That really opened a lot of professional doors, conversations with the industry players. It was a real game changer for me!
Cinematic Comedy Narratives in XR
E. S. - Cinema works by taking many different points-of-view, of the entire audience, and funneling them all down through the “vehicle” of the protagonist. And your emotional catharsis comes via that vehicle. But part of why that works is that “point of view” in a film is constructed and suggested out of many decisions and many moments, camera placement, lens selection, movement and editing all construct a psychological point of view that is fluid and subtle and definitely NOT constricted to an absolute place. But in VR, point-of-view is concrete and absolute, a fixed point in space, and because of that, many of the cinematic techniques just fall flat trying to use them in VR, or work much less well than you intend. VR narratives are so often boring, when they would have worked fine in traditional film, or they just lack the emotional punch you expected because that concrete point of view is undermining that “funnel” relationship with a protagonist. The catharsis happens to him or her, over there, instead of to YOU. So what I was trying to work on was ways of building psychological point of view in VR, a relationship with a protagonist that is very cinematic, and works despite of the fact that you also have a physical point of view to work with. Balancing those is the whole trick, or rather making both of these types of points-of-view meaningful simultaneously on a moment-by-moment basis in your narrative.
KAIJU CONFIDENTIAL at Cannes XR 2020
E. S. - I'm very proud of KAIJU CONFIDENTIAL (selected this year at Cannes XR Virtual - link), which premiered last year at Sundance and was nominated for an Annie Award early this year. It was written by a very funny writer who had no experience in XR. It was one of multiple scripts that was being considered for VR piece by a traditional animation company, ShadowMachine (link), that makes amazing animated television and didn't have deep experience in immersive. But they saw my piece EXTRAVAGANZA which is comedic and brought me in to direct and lead the way conceptually. And this particular piece had a huge opportunity for VR built into the script, because it’s fundamental insight into the characters had to do with size and scale -- these big scary monsters are actually psychologically small with their pettiness and insecurities -- which was an ingredient I could exploit especially well in VR. And most importantly, exploit moment-by-moment and change in unison with our hero to further illuminate his decisions, the power relationships, and the pivot points in the movie.
E. S. - KAIJU was an opportunity to take my ideas about movement and scale that I've been working on and really see if they are applicable on more than just one project. I had very specific rules to follow, I want the movement of the viewer to feel like it's inevitable. It's not even a question. Why did I move across the city? It happened so seamlessly with the content of the scene and the dynamics of the character that you don't even notice that you didn't control it. That’s that balance between your agency and the psychological point-of-view of the protagonist that I’m talking about. It’s really an editorial decision; in cinema, the choice of what shot to cut to and from what camera placement and what lens is seamless in a good film, it’s inevitable and timed in such a way that the new information of the new shot is presented exactly as the tension of the viewer grows and they just become aware of even wanting to see that new information As an editor you either work with the audience’s expectation or against it but you are always balanced on that knife’s edge so that the change of shots becomes completely invisible.
E. S. - So in VR I’m doing that with movement, editing space, so that it’s your own movement and size that happens on that knife’s edge of awareness, and your connection with the protagonist. In KAIJU CONFIDENTIAL, you start at human scale. Your initial relationship is the one that makes sense, because it’s with a human character who's hiding from the big and scary monster out the window This is how you normally see a Godzilla-type character. But after the moment of the character revelation where this particular Godzilla is picking his nose and you're like, “what? I've never seen Godzilla pick his nose”. That is the moment that the viewer transitions into being at monster scale. And in fact, you’re slightly larger than the monster. He's not very scary now. That is a VR-only technique, scale. It doesn't exist in cinema in this way, because a wide shot or a close up does not communicate the size of the viewer at all. It has to do with the object, not the subject. But in VR, the subject is primary when you change size or place. That's the first thing you notice. And then the object is secondary. This inversion of subject and object, it is a difficult thing to work with if you're coming from a film, it's not your expectation. The solution to that is to tightly tie those transitions to your character so that you don't even notice it happened, that physical transformation is so in synch with the character’s journey that it’s invisible.
AJAX ALL POWERFUL at Tribeca Immersive and Annecy 2020
E. S. - But there is a problem with using a non-interactive platform like linear 360 video when the storytelling is so delicately attached to the visitor location’s in space; the problem is that I can’t tell where you are looking at any given time in 360 video, so how can I seamless make movements happen and punchline’s land in tune with your movement? I can’t. I can put in the visual cues to direct your attention, but timing becomes a guess. And that's the difference with my current project AJAX ALL POWERFUL, which is made in a game engine. It takes the same visual language of size, scale and movement as KAIJU CONFIDENTIAL, but it adds in the ingredient of interactivity and timing. The visitor is never aware that they have any control over the world at all. But the pace of the movie is actually highly related to your view, things always just happen as your eyes land on them, and this includes when you move and when you transition in the space.
E. S. - Comedy is especially good for work like this, because pace is everything in comedy. For instance, in KAIJU CONFIDENTIAL when you realize that the character of MegaHydra has come into the city and is messing around on Grigon’s turf, that's only funny if you experience it as a sudden revelation, a shift in sound and tone, and energy. It really depends on directing attention and time very finely. Making it interactive behind the scenes makes the storyworld able to accommodate different people and different paces. The interactivity actually makes the story more similar across different viewers, which is the opposite of the approach many people seem to rake reflectively to something interactive. The usual project is about making multiple endings or multiple paths or decisions so that people’s experiences are different from each other’s, but that’s taking a page from games instead of cinema. I think if we want cinematic VR -- emotional catharsis in the cinematic model -- then interactivity is actually a key ingredient to doing so, even though it’s entirely behind the scenes.
E. S. - We’re in the official selection at Tribeca Immersive (link) and XR@Annecy (link). If there's ever an installation later this year, it would be amazing to exhibit it as it's intended but for now no one has actually seen it. AJAX is similar in many ways to KAIJU in terms of the comedy, the tone and pacing and the primacy of user scale relative to the world. Why I even started developing a project about a genie was because very specifically, genies live in lamps and then they come out of them, which is the opportunity for scale and nested spaces.
E. S. - That was the entry point. In AJAX, there's an interesting paradox when it comes to genies, they are powerful, yet they're powerless, at the mercy of the wishes of their master. So I wanted to express that at the different sizes, inside the lamp, then outside in the real world, and then finding opportunities in the story for our genie to become huge, towering over everything. And at each of these character scales, the visitor has their own scale, they get bigger and smaller but not always in unison with the characters. In that way the power between the characters will be illuminated or inverted because of their size. The big one might be more powerful or less powerful. It's all about size and power. And that was the framework that I started writing inside of, before I had any sort of plot or character details. It was starting from almost an architectural idea first of these nested spaces and the building character and story to inhabit that space in a way that was perfectly matched.
E. S. - I was lucky enough to take this idea to VeeR, which is a Chinese immersive distribution platform, which was just beginning to fund original content. The project continued to grow over time, more than I expected initially, as we realized that the most exciting hardware out there is the Oculus Quest. The finished piece is 15 minutes, give or take a slight variation in running time for different visitors, and built specifically for Quest, starring Henry Winkler and Chris Parnell.
Presenting XR pieces in the world after Covid-19
E. S. - I definitely have worries about that. Asking people to come back into a movie theater and watch a film, that's one thing. But asking people to go into a VR installation and put something on their face that someone else was just wearing, that is another level of intimacy that I do believe is a problem. The cancellation of film festivals is definitely a big deal for somebody like me, since so much of this business happens at festivals. Professionally, my next projects depend on people seeing something of mine and then being interested in financing something new because they believe there's some demand for content for a wide audience, so it’s fairly precarious.
E. S. - I'm very, very excited about the Quest. It was a delight to develop a project for it. Seeing the power of that headset, it's reinvigorated my enthusiasm for the hardware.
E. S. I’ve developed some prototypes and written some scripts for AR. The big problem that AR for mobile phones solves is not needing a headset, so you can reach a bigger audience. But narratively you have to deal with the space the user occupies. The stories I've been telling so far are not in your living room. I would love to see robust hardware that can transport you from your living room seamlessly into a completely virtual environment. Because I think that's right for storytelling to get portals that open in your room that take you to somewhere else, but for now you are still contending with the physical space of the user as the stage for your entire story. That is a conceptual obstacle for me that won’t disappear until these new magic AR glasses arrive.