XRMust_ProjectImmerse_Still01.jpg
May 20th, 2021 | by Agnese Pietrobon

"To open up an environment that encourages debate and discussion and experimentation" - Lance Weiler, Char Simpson (PROJECT IMMERSE)

Project Immerse is a virtual experience that revolutionizes our use of classic tools like Zoom to touch new frontiers of storytelling

What is innovation? There is an open debate around this word moved by the new possibilities offered by the media and by the choices made in the festival circuits.

Generalizing, for some, innovation is a concept tied exclusively to cutting-edge technologies. These types of innovators break new ground - the Marco Polos of technology - but what they create with that innovation is often inaccessible to the public. Works for a fortunate elite that find their ideal place at trade shows rather than festivals, and especially not at festivals that call for a more mainstream audience.

Then there's another kind of innovation - and it's the kind of innovation that I honestly appreciate the most because it doesn't just speak to the elite, but it tries to make itself known to anyone who is eager or curious enough to see for themselves where this world is going and what we can do using that cutting-edge technology. It's an innovation that's intrinsically tied to storytelling and how popular devices are used in innovative, sometimes surprising ways.

At CPH:DOX 2021, the Inter:Active section was a place that addressed this second interpretation of innovation in a very broad and boldly accessible way. It did so particularly in relation to documentaries, the core of this fine festival that took place in Copenhagen and was accessible online from April 21 to May 12.

Project Immerse at CPH:DOX 2021

One of the works that stood out at CPH:DOX for its innovative storytelling was certainly Project Immerse, a “deepfake paranoid thriller that places participants within a virtual experience that mixes story, play, and web pervasive technologies” (x).

This project works with and talks about one of the most advanced - and somewhat scary - technologies, artificial intelligence, and in fact has strong ties to academia, with key collaborators being the Columbia University School of the Arts’ Digital Storytelling Lab, Andrew Saltzman Institute of War & Peace Studies, the Brown Institute for Media Innovation and the Teacher’s College Creative Technology Program.

XRMust_ProjectImmerse_Still02.jpg

Because of this, the way Project Immerse delivers these same technologies to the public is through a medium we've all become familiar with over the past year, namely Zoom, of which Project Immerse uses the most common features in a highly dynamic and narrative way.

In addition to Zoom, another web tool comes to support the story and the actions of the users: the collaborative whiteboard platform Miro, which I was mostly unfamiliar with, but which has become a popular tool in places like universities. The most surprising thing, though? 90% of Project Immerse, as the authors say, is created by artificial intelligence. And even if you know that, and you're open-minded to the idea, it's still a shocking thing to face when you're part of the audience experiencing this work... and an even harder concept to process after you're done.

I reached out to Lance Weiler, co-creator, co-writer and director of Project Immerse, and Char Simpson, co-writer, to get a better understanding of how this piece was created and learn more about what goes on behind the scenes. Here's what they told me.

A serendipitous meeting of minds

LANCE WEILER - Around this time last year, I was doing a show called Where there's smoke, for which I used Zoom and Miro, and I had also been running speculative design sessions called From the Futures along with The digital storytelling lab. In a conversation I had at the time with Nick Fortugno and Zeke Zelker, we started talking about the idea of working on something about misinformation and deception. Those topics had been very popular over the last few years - fake news, the challenges presented by social media in particular - and we just felt it would be interesting to explore them using an approach similar to Where there's smoke - which has theatrical potential and uses common productivity tools to subvert them and create something interesting with them.

We put together an initial application for funding through the Brown Institute, which was awarded and that kicked things off. We were going to do it regardless of the funding, though, because I really think Project Immerse addresses an interesting topic and we're just scratching the surface of what's possible in terms of virtual immersion. Then Char came to one of the shows I was doing for Where There's Smoke and contacted me afterwards.

CHAR SIMPSON - I had a very serendipitous connection to Where there's smoke. A couple of collaborators I worked with on another project called me to check it out. I did and I loved it. So I emailed Lance, said "we should talk" and we did. And that's how we started working together.

On radicalization: falling down the rabbit hole together

L. – I recently read a New York Times article titled “Is QAnon the Most Dangerous Conspiracy Theory of the 21st Century?” and its subtitle caught my attention: “It’s a collaborative fiction built on wild speculation that hardens into reality”. In a way, this is at the heart of the Project Immerse: how the way we shape narratives, the concept of apophenia (a/n: the human tendency to perceive meaningful connections between unrelated things), as well as our desire to recognize patterns in what we read, do, and watch can lead to radicalization and cause us to fall down these... rabbit holes.

We actually started with a radicalization storyline for the pilot. We were really fascinated by QAnon and we went from there about a year or so ago. Then, when there was the attack on Capitol Hill, everything somehow came to the surface and the radicalization storyline became even more poignant. It turned out to be a real thing, something that people and the news kept talking about, because we had just faced the biggest threat to national security and it was deeply tied to the radicalization of ordinary people living inside our country. And that knowledge, in early 2021, made this piece incredibly relevant to us.

C. - Project Immerse demonstrates how susceptible everyone is to falling down conspiratorial rabbit holes. There's this notion, especially around the US election and the invasion of the Capitol, that only a specific group of people could be caught up in something like QAnon. But the fact is ... the group at the Capitol riot was incredibly diverse and dynamic. They came from a variety of different backgrounds. The risk is universal. Falling down a rabbit hole is so engaging! It's an almost theatrical experience, something that provides community, and the thrill of group speculation. A shared conspiracy is engaging in many ways, it has the potential to generate power and violence. When wild speculation leads to action, events like the storming of the Capitol unfortunately follow.

The importance of creating a safe space for participants

C. - Experiences like Project Immerse allow you to create a safe space for participants to connect and experiment with things that are potentially harmful.

Our lives are often spent trying to cope with traumatic experiences. Another piece I co-wrote with an AI called Alexa, Call Mom! (presented at the Tribeca Festival last year) is a single-user experience where Alexa acts as a spiritual medium and connects you to a dead mother character. Our team was really interested in Alexa beyond her voice assistant abilities. We explored themes of possession, feminism, and how capitalism commodifies death and grief. Those are really intense concepts for all of us and Alexa, Call Mom! provided a safe-space for users to explore these concepts through a horror/comedy lens. Alexa, Call Mom! and Project Immerse share a mission: You can give in, let yourself get pulled down this rabbit hole and speculate. It creates a really fluid educational space for learning about yourself and connecting with others.

XRMust_ProjectImmerse_Still04.jpg

On branching storylines, microstories and audience participation

C. - My background is primarily in branching narrative experiences. I write Choose Your Own Adventure books for school-aged children and create immersive experiences for all ages. What's really interesting to me is the way storytelling is changing: non-linear storytelling is becoming the norm. It feels like more readers and users want to have a say in where a story goes. They want the chance to choose a path...

I think of Project Immerse in the same way. We have three paths that we feed to the audience to push them to speculate more, but every text, every image that the AI generates can actually send you to another corner of a much bigger conspiratorial story. During the show we entice people to keep moving and keep speculating: the AI is always providing you with these micro-stories that either derail or feed your theory.

L - The project has many layers. First, there is a collective exercise in sense-making. Audience members become storytellers. The pieces are there, but they need to be assembled and they are assembled collectively by a group of people who bring their own biases to the story, their own perspectives of the world. They look for patterns in different ways. This is how the work explores the idea of apophenia, and how we make sense of the world through stories.

Project Immerse is an immersive piece but a really participatory one, with no rails, unlike many other immersive pieces. It's a generative project that is different every time, based on the people who are there and where you fall in the experience.

It's a lot like the rabbit hole Char talked about. And because Miro is an infinite canvas, you can literally get lost in a way that you can't normally get on the web. The web is very standardized: you know where the menu is, where the sitemap is, you know how to use the back button. In Project Immerse you need to find new ways to tell where you are and coordinate with other people within the board space: this creates a whole other layer to the piece, which is about the physicality of the virtual space. It calls for the ability to move through a space where you're discovering different things and simultaneously being in a group in a Zoom breakout room full of people who are doing different things on the board at the same time. It's a piece that is extremely exploratory, in a way that narratives usually aren't. It borrows a bit from MMOs (massively multiplayer online) and sandbox games, but then has narrative threads that connect things.

The role of the Artificial Intelligence in Project Immerse

L. - We embrace the AI as a creative companion, building on the work we've done at the lab and pieces like Frankenstein AI, which I worked on a couple years ago, and Char’s Alexa, Call Mum!.

In Project Immerse, the AI is credited along with Char and myself as co-authors. Some nights, actually, the All takes all the credit! It works in a really non-linear and interesting way: it challenges us in terms of the story, in terms of what the aesthetic looks like, in terms of what the potential interaction is.

This piece is based on a creative process that augments artificial intelligence - something that I think we're going to see more and more of. That's why this piece is also a meta-piece, very much like a Mobius tape or a snake eating its own tail. You get lost in it, the same way you fall down those rabbit holes and become convinced of certain conspiracy theories and start to be attracted to a specific group of people.

Those are the initial steps towards radicalization because at the end of the day, it doesn't matter the group you join - QAnon or some other radical group. You just want to connect with people, feel that you're not alone in this world. So we're using some of these techniques, but also creating an arc of confusion and paranoia and eventually working towards something that's really about connection. That's why I think Project Immerse is somewhat unique in the way that it embraces the machine and uses it as a creative Muse, in shaping not only the story, but also the core of the experience and the interactions.

XRMust_ProjectImmerse_poster.jpg

The pros and cons of using accessible technical tools

C. – What’s so fascinating about Project Immerse is how pervasive the technology we’re using is. Everything - every app, tool, device we're using - is very, very accessible. A lot of the work on videos, images, sounds, was done on Lance’s mobile device. I've never programmed an AI before but using accessible platforms like runwayml gave me the freedom to experiment with different datasets and build our co-writer, NightFall.

A key principle for Project Immerse and within the Digital Storytelling Lab is that storytelling should drive the way we implement technology and not the other way around. These pervasive tools that allow you to create a deep fake in seconds create so many new roads for new types of storytelling and new types of deception, and fun interactive experiences. So on the one hand, this is all really exciting for storytellers. On the other...the threat of misinformation looms larger, which is part of the reason why we wanted to use these pervasive tools: to show the audience that you can create engaging interactive stories with these tools and to make them aware of how damaging they can be.

Behind the scenes: stage management in Project Immerse

C. - Both Lance and I have a theater background and there is a lot of stage management in mine.

Project Immerse, interestingly enough, plays out very much like a theater production. We have Kate Johnson, who is our stage manager and coordinator for everything Zoom, and Takashi Okada who is our set designer and art director, and I call the show and move us from section to section. The whole team coordinates scene changes, facilitates participant movement, and interactions. It has a live theater feel despite the fact that we are in Zoom.

When you're running an in-person immersive show everyone is there, so there are different kinds of challenges. Doing the same thing remotely has a very different feel. Time moves differently. I remember when we started testing Project Immerse, five minutes would go by in the blink of an eye. Now the show has found its rhythm and has a certain fluidity to it, and we have a shared understanding of what needs to happen next.

Project Immerse is a collective management effort. Additionally, many of us take on dual management and performance roles in the experience. An important insight from the stage management/performance perspective is how to simultaneously facilitate user connection and stay true to the story in a virtual space. Which I think is something that a lot of creatives and artists have been grappling with over this past year.

Connections of Project Immerse to Academia and education

L. - Project Immerse is really about the idea of building immersive learning experiences. We're shaping a new grammar of how that kind of experience works. A lot of people come to Project Immerse and say, "I've never experienced anything like this before", and other people who create tools and platforms say, "We've never seen them used like this!”.

So, this is uncharted ground, and when you're doing something uncharted, you're learning as you go along, and there are no rules associated with it, and that makes the journey really exciting.

At the same time, it creates a really interesting environment that allows you to take risks and be bold. We try to document everything we do and share the process in our environments. We weave it into our classrooms, for example: Project Immerse was running in our new media art class, in our digital storytelling classes. As we were making it, we pulled back the curtain on what it was and ran its various incarnations with students.

During my fall semester last year, students had to create adaptations that explored misinformation and deception. So within the class they were exploring things that were similar thematically to Project Immerse as well.

XRMust_ProjectImmerse_Still03.jpg

It's something that creates a really rich environment and I think the more holistic the approach, the more it allows you to discover things. It's a wonderful iterative process that's intertwined with the creative process - some of that iterative process comes out of design thinking and speculative design practice and then marries with play and MDA (Mechanics Dynamics Aesthetics). But we also weave in narrative design techniques.

So I think creating and maintaining the space for the learning environment is critical - you never know where amazing things are going to come from, and they often come from things that are broken. So the faster you can break something and the more hands you have on it, the more you learn.

All of this really connects Project Immerse to the university. So much so that now the theater department uses Miro for students and a series of plays. We're in a time where people are adapting to new directions. That's why I think a lot of the work we do both at the lab and with this project, and also what I'm doing with Where There's Smoke, is about pulling back the curtain on the process, allowing people to see how the levers are being pulled and really trying to open up an environment that encourages debate and discussion and experimentation.

C. - I'm also thinking about bringing Project Immerse to a teaching initiative I work with in Los Angeles. A lot of the middle and high school kids I work with have a different way of looking at storytelling. The internet is a huge part of their lives, so the way they take in information is primarily non-linear. The way they create stories and relate to stories is completely different than the way I related to stories when I was their age. I'm seeing a lot of students go beyond linear storytelling - in terms of having a beginning, middle, and end - and become frustrated with the models that came before this time.

That's why I think projects like Project Immerse, which allow the audience to become the creators, have really encouraged them to create work on their own. As a teacher, it's important for me to think about how I can push the kids I work with to create stories that are really cutting edge. Some of the students create things that I honestly never even imagined! And it's because they understand the grammar and movement of non-linear storytelling better than I do.

On the future of Project Immerse

L. - We are actively developing, iterating and building the project and will be taking it through the summer. At the same time we'll be working on three more episodes, so we'll have a total of four episodes. We are also developing an alternate reality game that will be released through NFT this summer.

C. - As we move forward and continue to create episodes of Project Immerse, we're going to start working with experts in different fields and releasing all the datasets we're using.

Whenever you use AI, you have to be doubly aware of what you're creating. Lance and I are human, we're not perfect, but in a lot of ways the story wouldn't be as accessible as it is now if it weren't for us having a really critical eye in terms of what the AI is putting out.

Whenever you're working with material that's so pervasive, it always signals a certain kind of darkness, but I think taking a match to that darkness and smoking out some of the real-life horrors provides a space for people to talk openly about their experience. Thus sparking genuine discussion and community awareness. This is the kind of art that we need to create.


Project Immerse is a work created, written and directed by Lance Weiler; co-created by Nick Fortugno, who was also the narrative designer for the piece; co-written by Char Simpson; co-created and produced by Zeke Zelker… and co-created with an AI.

https://lanceweiler.com/project-immerse/

Share article

Related posts

XRMust_PrisonX_Still02.jpg
February 22, 2021

“VR as a tool to democratize animation” – Violeta Ayala, Roly Elias, Alap Parikh (Prison X)

Prison X is an immersive Andean mythological play that, by working on representation, brings the user face-to-face with a different way of building stories and a culturally vibrant experience.

XRMust_PixelRipped1995.jpg
April 24, 2020

PIXEL RIPPED 1995 is finally out on Oculus and Steam VR

In these times of stay-at-home confinement, we may not go out as much as we'd like, but at least we can invite you to a multi-dimensional VR trip back in time to 1995, an era that planted the roots o…

XRMust_ThePassengers_Still01.jpg
May 25, 2021

"Animation, documentary or fiction... immersive experiences are more and more hybrid!" - Oriane Hurard (Atlas V)

With a double career as a festival programmer (notably the GIFF in Geneva and I LOVE TRANSMEDIA) and producer, Oriane Hurard has spent the last ten years at the heart of new writing until winning the…