“I realised that, in a virtual environment, I needed to give the dancers more conceptual content” - Gilles Jobin (COSMOGONY)
I spoke with Gilles Jobin, dancer, choreographer and technologist, about his company’s extraordinary work in the live performance space. Compagnie Gilles Jobin is working on a VR project called SUNSET MOTEL with Swiss illustrator, Thomas Ott, that is currently in development for 2023. The project employs motion capture technology combined with traditional illustrations to create a unique form of recorded VR experience.
I asked Gilles 11 questions to learn more about this project and his company’s past experimentations and successes. If you can’t dive into the full interview, here are four quick takeaways:
- Digital shows can be adapted and changed much faster than traditional theatre shows, something that cannot be done in traditional filmmaking.
- With regards to the production pipeline, Giles needs all his assets, especially the avatars, near completion at the beginning of his creation process. Together with a mostly complete setting, these assets can then be used by all the creative parties (dancers, musicians, technical crew) in real time for creative collaboration and experimentation.
- Gilles states the need for 3D artists who can work with live performers. His work isn’t just on the bleeding edge of experimentation from the performer’s point of view, it is also creating new types careers in theatre.
- This new hybrid form of entertainment will not replace traditional theatre. It is a new form of entertainment in its own right, that can and should be scheduled alongside traditional productions. It will not replace theatre. It is an additional offering.
What was the inspiration behind choosing COSMOGONY as the title of your project?
Gilles Jobin - Well, peoples of the world, they each have their own COSMOGONY. They have a story about the creation of the universe, but [it] is also the study of the universe, like the real one, the physical one. So it's something a little bit in between. Also, when you choose the title, you need to have [one] that is kind of generic enough, not too precise, you know, I [am] playing [with] technology and really prototyping. You [are] never sure what you're gonna get.
COSMOGONY is a live performance featuring two male and one female dancers whose movements are normally captured live in the studio of the Gilles Jobin’s company in Geneva and projected in real time in the Plaza, but in this occasion, the capturing of the dancers and digital projection will be at the Cinema le Plaza, so the audience can see the both sides of the work.-
Your dancers have physical markers on their bodies that are read by cameras mounted above them, does it inhibit their movement in any way? If so, how do they adapt their choreography to accommodate these devices?
Gilles Jobin - We use an optical system. We have about 40 cameras, from Qualisys. The dancers, they have markers… they [are] just reflecting the infrared actually. And the cameras catch it from all different angles and create a skeleton. On top of the skeleton, you can base another avatar. We connect this directly, live, into Unity. When I move under the mocap system, then my avatar is being animated into Unity. In real time. And then from there I can have a real time output. It's basically like online gaming: you have your own PlayStation at home. You generate the game locally and then you connect to a server, and then that's how you grab the position of the other players in the game and you share your own position. So basically that's what we do to do live digital performances from our studio into the world
Gilles Jobin - The venue that invites COSMOGONY must provide a good PC, we send them the build of the piece beforehand, they generate the build locally on their PC and they connect the projector or another video output. It requires a stable internet connection to connect to the server to grab the mocap data live from our Geneva studio. So that's why it's very light and, and there is no lagging with an HD quality screening. We developed a tool for this technology that allows us to invite any motion capture system into a 3D space such as Unity. So you can be from South Africa in, uh, a Rococo suit dancing with (dancer) Susana here in Geneva, and somebody somewhere else dancing together but distant from thousands of miles away. This is feasible today and opens many creative possibilities.
Gilles Jobin - Now, in terms of the dancers. I mean, of course they have round markers on their bodies, so that's something to take into account when they move. The mocap system is so precise, that it'll catch everything they do, every little detail. The only thing maybe is that if you stay still, it looks very still… You have to keep the avatar alive otherwise they appear to be frozen. Then there's the question with the hands, don't do movement that has to do with fingers because we don't have fingers... And then the occlusion. [Some body movements] can make occlusions that's gonna create some problems.
Gilles Jobin - There's also a difference between real time and recording. When you're recording something, you can correct [mistakes] afterwards. But when in real time, of course not, because it [has] a momentum and you can accept some glitches in the real time that you wouldn't when it was recording. Also the quality of the optical system… if we're talking about the Rokoko suits or a Perception Neurons suit, that's a different ball game because they are less precise and slide in space. But they can come handy for many different situations. You have to understand the mocap system, and adapt it to your project. Each project has its own MoCap system.
What was the IRL audience reaction to the live shows (COSMOGONY and VIRTUAL CROSSINGS)?
Gilles explained that with COSMOGONY, the live choreography done under the optical system wasn’t designed to be viewed as a performance itself. The company gave the audience access to the physical dancers, in order to prove that the virtual show was done live, in real time.
Gilles Jobin - It shouldn’t matter but it does. It matters because there is the big question [regarding the] synchronisation of audience and performers. I think the first football game live on TV in color was in 1972. So it's been a while that football people accept the fact that they are playing live, but they're watching [from across] the world and they're watching live. [The viewer can get quite emotional] and actually it happened 30 seconds before, there is a buffer. But for us, synchronized in Europe it is live. Maybe you noticed that today, live sport transmission is not synchronized anymore with the different platforms, and your neighbor will shout before you to a goal! I think it is a flaw that somehow changed this precious relation between actors and spectators. A typical glitch that software engineers might not regard as problematic when it is very much conceptually!
Gilles Jobin - The synchronicity between the audience and the performer is one of the main interests of Gilles’ work. This connection and sense of immediacy between the performer and the audience, is what Gilles seems to be experimenting with in all his projects.
Gilles Jobin - We did another project, VIRTUAL CROSSINGS, a collaboration between different artists, visual artists, dancers, musicians, 3D artists. We made that project knowing that the audience was going to be present, so that was totally different from COSMOGONY that was created for a 2D screen. In this case, the idea was to put the rig inside an old cinema called Le Plaza in Geneva, bring our motion capture cameras and create a collaborative piece in 2 weeks, in real time for an in-presence audience.
Born during the pandemic, VIRTUAL CROSSINGS was imagined in order to enable movement artists and digital artists to connect remotely and meet in the same virtual space without the constraints of geographical borders, and to collaborate in real time…Thanks to a sophisticated motion capture device installed in the centre of the cinema, the dancers’ avatars interact live on the walls and screens of the Plaza.-
Gilles described a moment where his dancers communicated to him that they felt somewhat lost, that their intentions were unclear to the live audience watching them.
Gilles Jobin - I [realised] I needed to give [the dancers] more conceptual content, they felt during the process a bit lost and very exposed. But I explained to them that the audience was at first watching them trying to understand the system, but after a while they switched their attention to the screen and the result, them, being multiplied as avatars on many screens. They were not alone and exposed on stage, they were avatars on the screens. It is like a deal is being established between the dancers on stage and the audience. The audience understood the system and watched the result, dancer and audience both project themselves equally in the work. It was very nice to see this connection happening and how the audience was with the dancers. Also how human and organic the piece resulted to be, full of human emotion…
What is the future of this type of hybrid live IRL- digital performance genre? Where do you think we will be as an industry 5 years from now?
Gilles Jobin - We started to realize that the system we use for the real time performances is also very good to do recorded VR projects. One of the problems with motion capture is that it's expensive. That means the tech is inaccessible for most independent artists… [When] you're gonna rent a [motion capture] studio for one day, maybe two, you don't have time to do any research in movement. We are dancers, that is so frustrating. You know, when I create a dance piece in my dance studio, I spend some days trying out, creating meaningful combinations between the bodies… I do not go out with a definite plan, I need time in a situation with the dancers for them to become creative, develop in movement. Too often in dance-tech projects, dancers are left for the last minute. When finally the tech is running they have not time to develop creatively. Big mistake! Dancers are problem solvers! Most of the digital pieces you will see involving dance are lacking time for the choreography to develop. It is partly due to the ignorance of the industry of the power of the choreographic language as a tool to create meaning… Particularly efficient in VR! We're gonna do [COSMOGONY] in 2 different cities in Italy, in three different festivals simultaneously.
Gilles Jobin - We broadcast it live on Geneva local public TV. We can have two or three different performances on the same day in different cities. So [this] is the beginning... but you know, as a performer, you could feel that it could be frustrating because we are not travelling anymore to perform. Today we are living this reality, performing tonight in El Cairo and next week in Milano. Having a show at 20pm in Europe and a gig in Milan at 1am because of the time difference! We call it the new jet lag! And sleeping in our beds at night… We call this #tourwithouttraveling We don't travel anymore, but we're still touring internationally. In 2019 we were touring like crazy, we spent approximately 90 tons of C02 in economy flight. In 20-21-22 we have not travelled but have as much international diffusion, but our carbon emission for touring are close to none. If we had toured in presence at the same level than 2019 that is around 300 tons of C02 that we didn't spend in 3 years…
When in pre-production for a new show, what is your planning process? Do you work with a concept first, then choose the technology, how many dancers you need? All these answers impose restrictions, I assume.
Gilles Jobin - We had to change the way we work. I started with VR around 2016 and I have only done digital pieces since. Covid came and we realised that we did a lot of different projects in AR in VR, real time, live collaboration, so we had the tools around us to try something else, going full digital. Now we are in a new moment, where we can play with all those tools and develop more. We start to know better what we want in advance so we can prepare and organise the time in the studio for dancers to spend time creatively in the system… Now for instance, I don't want to start a project [if] I don't have my avatars! It's always one of the problems we have, the avatars. They always come so late… and it's like a nightmare. So now we say, first we get the avatar, then we get the set, and then we can start to work!
And your reason for having the avatars first is that for characterization? So that your dancers can consider the constraints of the actual design of the character and its movement?
Gilles Jobin - No, I mean, you know, I'm an auteur. I'm not like an illustrator or a theatre director [who] has… the text [who] has to kind of make a stage version of a preexisting story. For me, I have to invent everything from scratch, and dance is mostly non-verbal, more abstract. It is about making the movement, the space between the dancers meaningful. I need the avatars because they are the cast. Imagine casting actors with no parts to play because the text is not written yet…
When in rehearsals, how is your approach to directing and choreography-design influenced by the new technology? How is the dancers' preparation in reherasals changed?
Gilles Jobin - The stage is a limited space and there's lot of constraints. One of the most constraint spaces to create [is the] traditional stage with the public in front, the 4th wall. But inside those constraints we find solutions to create very different stories in the same space. We are very much used to working with constraints in contemporary dance, we are live in front of a live audience, synchronized. That is already an enormous constraint. For me, tech is just new and different constraints. The difficulties [are] to find what those constraints are when theoretically all is possible.
Gilles uses the analogy of football, saying football isn’t improvised, there are rules, a strategy, yet the players must respond to changes in the moment. Gilles doesn’t choreograph in the sense that he teaches movements, but more as a guide or a coach.
Gilles Jobin - So many people from the performing arts [don’t know] that these kinds of technologies are interesting to us because you can use them for many different outcomes. You know, when you tour in dance… maybe two weeks [into a] tour you start to reassess the piece, you change some details. It is only after a few gigs that the piece takes its final form. A live piece gets better every time you perform it. Digital is quite similar. I say, oh, I don't like the color of this avatar or the shape of this mountain, we can change it. It's feasible, you know, it's not like [traditional] cinema [where it] is impossible to reshoot a scene.
What you just described, that is immensely exciting. And I think from a directorial point of view, that must be so organic and instinctive.
Gilles Jobin - Yes. But you see how you need to get organized to get to this point? You have to define new priorities in how you would [like to] do a VR piece. Traditionally, in an animation or like a movie, you get all the good stuff at the very end. [For me] there's some elements we need from the start because otherwise I can do nothing creatively. Too often in tech an artist spends more time being a producer and technologist than a creative artist. We want to give more space to the interpreter inside the construction of the story. We need time to expand during rehearsals, also test the wrong directions… The tech is available now so we can do it today. It's not that difficult. It's just a question of organisation. Having our own MoCap studio available as much as we need is definitely helpful…
Your advice in general to other choreographers, directors or performers wanting to move/pivot their careers into this field?
Gilles Jobin - Well, first they need to see some other projects, try to know a bit about the tech… a minimum understanding of the tech. Do your homework. You know, what is a game engine? What are the different types of motion capture, how to integrate it into a system. Uh, what will be your outcomes, VR or on a PC, a video game, an AR piece? If you think about it, it is the same on stage. I mean, I was a light technician. I worked as a technician when I didn't have enough [work as] a dancer. I worked as a technician so I could learn. You know, I always prefered to swipe on the theatre floor than in a supermarket store. This is how I really became a professional choreographer. Knowing the mechanics of the theatre was my best ever move. Otherwise you [will be] totally lost.
Gilles Jobin - Technical knowledge is necessary for you to organise the team, the pipeline of production. What is the first thing to do? Do you need the avatar first? Do you need to have a motion capture system or is [it] gonna be animated? Or are you gonna use video? Who are the collaborators that you need for your project? Many counter intuitive questions arise that can be tricky to resolve. Follow workshops where you can learn… be aware of what the opportunities are and start to get into it. But most of all find your own “geek” to do the link between the real world and the virtual world. For me Camilo de Martino, our technologist, was essential from the start in developing all our pieces, I could not have done it without him.
Gilles Jobin - Then get close to maybe your local university where you know that they do some motion capture stuff or like maybe, a studio. Try to get access. [Don’t be] only theoretical… Also it is very important to develop a common language with the tech because that's the biggest problem that you might find is that they don't understand what you're talking about and you neither... [It’s] also a learning curve starting from where you meet each other. If you have a good team, stick to it! I’ve got a dream team!
What was your motivation behind creating the DanceTrail App?
Gilles Jobin - DANCE TRAIL is an app that allows you to place dancers in the space. It is fun and playful, we created it for the Sundance Festival… It came out discreetly but then it caught on its own little life. People are using it, quite a few festivals are inviting it. It's free, really easy to use and it is creative as people can make video, photos. It is fun! "DANCE TRAIL is an augmented reality piece in the form of a mobile application that allows users, through a smartphone or tablet, to invite virtual dancers into the real world… take an augmented dance journey in the streets, in the fields or in the palm of [your] hand”. It is available on Android and IOS versions.
Will these new technological performances replace live theatre?
Gilles Jobin - Of course not because it's something else. It's a different thing. It's a different experience. In a dance house somewhere in, uh, Australia, for instance, they can have COSMOGONY very cheap because it costs very cheap. They don't have [to budget for] hotel [accommodation]. There's no travel, no pickup at the airport. For us it is only one day of work, a week to go to Australia! They just need a bit of equipment to present it, a good gamer PC, an internet connection and a projector to make it happen. And they have a live show from Switzerland. That's it. International, no carbon emission, it is even good for their karma!
Gilles Jobin - Probably the next show on their program will be a company from, you know, China or… local from Australia. I think we're gonna have more hybrid programs. Different possibilities and accessibility.
What motivates you to continue to push the boundaries of hybrid IRL dance-digital/technology experiences? Why do it?
Gilles Jobin - (He smiles) I don't know. I mean, I cannot do like twice the same, I guess. I have a lot of difficulty repeating [something]. But now we are changing a little bit. Now we want to explore more [of] what we have… put ourselves in a place where we know our possibilities, our equipment, all those tracks we didn't really explore.
What’s next for your company?
Gilles Jobin - We are working on a VR project called SUNSET MOTEL with Swiss illustrator, Thomas Ott, that is currently in development for 2023. He's a “scratch” artist. So he starts with a black sheet and then he scratches. So he takes out dark matter to create light, which is contrary [to what] you do when you draw normally… it's very dark and… really nice, funny also. It is really kind of amazing. We found a solution for his scratching to work in 3D and in movement. The VR piece is going to be recorded, but it's gonna be constructed almost like a theatre piece, with a lot of situations created with the dancers in the studio because each character has their back stories and they're in this motel… there's not necessarily a narrative, but there is a backstory for each character. We hope to break ground with our method to create a story. Thomas Ott’s world like mine has no words, only images to create meaning…
That sounds so exciting. (I was given a sneak peek of their rehearsals and the illustrator’s work for this specific project. It is gorgeous!)
Gilles Jobin - We also have a creation that's gonna be on stage in 2024, which will have live motion capture and dance on stage. That's gonna be a proper stage piece. Also we're starting to have resident artists at our studio. We aim at creating a community, an ecosystem around us in Geneva, reaching out to people doing this kind of digital stuff because we need more artists, more technicians, we need more people, more women in the field. We need 3D artists, interested in live performances. These are the new jobs for the performing arts.