Company Post

TouchDesigner Drives Real-Time Digital Game-Mechanics in Foxconn Frequency

Foxconn Frequency (no.3) - For Three Visibly Chinese Performers is a work of "algorithmic theatre" that incorporates real-time game mechanics, piano pedagogy, 3D-printing, and the poetry of former Foxconn worker Xu Lizhi.

Three performers work with and against each other as they move through a series of testings and mini-games sourced from piano training. They succeed and fail in real-time, forced to correct their mistakes before they can continue. In each performance, 3D-printers output an object documenting the performer's competency over time.

Fully automated and generative, the work is different every performance, ranging from 50 minutes to 80 minutes: an experiment in digital game-mechanics as a way of meaning-making in the performing arts.

TouchDesigner was used in all aspects of the design and development process: to program all the game mechanics, the audio/visual feedback for the performers, and what the audience sees.

This is the last iteration in the Foxconn Frequency series from Hong Kong Exile, the Vancouver-based interdisciplinary arts company. We spoke with project lead Remy Siu who was responsible for all the TouchDesigner programing to learn more about the making of as well as the experience of staging and performing such an impressively original and ambitious work. Thank-you for talking to us Remy!

Derivative: First, congratulations again on a really relevant work and its ingenious execution! Goosebumps! We understand it has been nominated for the Gaudeamus Award 2019? Can you tell us please about this award and how it is awarded?

Remy Siu: Thanks!! This answer is straight from Wikipedia: "The Gaudeamus International Composers Award is a European award issued by the Gaudeamus Foundation. The prize is awarded at an annual ceremony in the Netherlands to a young composer competing in the highly competitive Gaudeamus Music Week.

The Gaudeamus Foundation had held an annual music week of Dutch compositions since 1947, alternating with an international competition until 1959, from which time they became fully international."

Every year there is a submission of works that usually involve a composer submitting a musical score. Since these works don't have a traditional score--everything lives in software--I actually submitted screenshots of the TouchDesigner network as my "paper score."To my surprise, the jury responded well to that.

Derivative: Can you tell us a little bit about your background? You seem to be comfortable actually doing many parts of the projects – you have many roles…

Remy Siu: My early background is in writing fiction. I studied music composition and film at Simon Fraser University Contemporary Arts. I was very lucky to go to that school, as it allowed me to also take part in experimental theatre productions and contemporary dance shows.

I started down the creative coding path after working on several performances where I performed live electronic sound scores. In most of those processes, the lighting came at the last moment. I desired to have the same access and flexibility with the lights as I did with the sound, so I began exploring different options like Processing, OpenFrameworks, VVVV, and Isadora. I used all of them at one point or another. (Isadora, the most).

At the time, because I didn't have a good understanding of DMX, I bypassed traditional lighting and only used "digital" projection light. My first works with projection lighting were with my partner--Mahaila Patterson O'Brien.

Today, I've mostly stuck with projection light as the main lighting source for my works and my collaborations.

Since I work in the experimental performing arts, our team is often small, so everyone on the team has to be comfortable filling multiple roles. For most of my work, I have to be comfortable leading a project from beginning to end, working in digital audio, western music notation (if needed), 3D / 2D, general programming, budgeting, tour logistics, interfacing with different kinds of hardware -- all of this was learned through process and experience! (And great online learning resources!)

With each project, I try to do something I haven't done before--or simply--don't know how to do. It's rewarding to pick up new skills and techniques. I'm a very restless person, so that may also be a part of wearing different hats on different projects.

(Howie Tsui). Algorithmic animation sequence, 5-channel video projection, Vancouver Art Gallery

Derivative: How did you first encountered TouchDesigner and decide to incorporate it in your work. Do you use it for a lot of your work and/or how does TD generally fit into your pipeline?

Remy Siu: I first encountered TouchDesigner through my research online about different tools I could use for my work. I *really* started using TD during a project called “Retainers of Anarchy” (premiered at the Vancouver Art Gallery) with my friend and collaborator, Howie Tsui.

Howie was working with animators to bring his figures to life, and wanted to create a 28 meter long digital scroll work. We wanted to have the camera zoom all the way into a room and all the way out into a landscape, so the assets had to be very high resolution. We also wanted to spatialized the sound along the wall, so that it felt like the sound was emanating from the moving figures. Finally, we wanted to have the project run in real-time, to create a sense of presence not possible with a “fixed” work.

“Retainers of Anarchy” has something like 20-30 2K+ video files playing simultaneously, and the whole thing is 7680x1080. After so much testing, TouchDesigner really was the tool for the job. The video playback is out of this world.

So, since 2016, I’ve been using TouchDesigner for all my work. I do almost everything in Touch, the visuals, real-time game mechanics, and these days, we’ve started to do digital audio synthesis and sound in Touch so that we can have more deep relationships with other parts of the work. (OSC was becoming a bottleneck.)

Derivative: Please tell us about Hong Kong Exile and how Foxconn came about.

Remy Siu: Hong Kong Exile is an interdisciplinary art company with three co-artistic directors from different disciplines. Natalie Tin Yin Gan is from contemporary dance and choreography, Milton Lim is from Theatre, and myself from music and creative coding. We take turns leading projects, with each successive work being less and less rooted in a particular discipline. We also focus on the Hong Kong diaspora and cultural politics in the era of globalization.

I don't remember the exact moment I combined piano pedagogy and mass manufacturing. I started working on the "Foxconn Frequency" series in 2013, before the death of poet and factory worker, Xu Lizhi. After his death, and the publishing of his poetry online, the series shifted to feature his work.

Initially, the set-up was designed for any "visibly Chinese" performer of any skill to perform, and the first entry in the series was a very simple prototype, trying to disconnect performer labour and the resulting sound.

In the second entry, Vicky Chow became involved. For people who aren't in the chamber/orchestra music world, Vicky Chow is a piano playing beast who is part of the "Bang on A Can" ensemble in New York. She is originally from Vancouver. The Western Front (an arts organization) paired us together to create a new work.

"Foxconn Frequency (no.2)" was the first time I introduced real-time game mechanics into my practice. It was a very simple mechanic. Play a piano phrase above a certain competency threshold or fail. If the performer fails, they have to do it again until they succeed.

In "Foxconn Frequency (no.3)", we took what was essentially a single player game and fleshed it out to a multiplayer game, expanding on a simple mechanic in many different ways.

I see these works acknowledging and honouring the poetry of Xu Lizhi. I'm not sure if I have the space here to write out everything about it, but one can go to my website to get more information!

Derivative: “What is the audience’s reaction?” was my first thought as I looked at the videos and imagined being in the room.

Remy Siu: The audience usually gets very anxious, but very engaged. Swings of joy, excitement, dread, and boredom. I often feel they are right there with the performers, experiencing real failure and success in real time!

Derivative: A fully automated and generative stage production is a pretty risky thing to take on is it not? Did you think at the onset that you could make it work successfully and how?!

Remy Siu: It *can* be risky! These works can theoretically last forever, if the performers get stuck in a passage. The other risk is that unlike traditional music composition, I don't have total control over the length of the sections, so it changes the way we dramaturge the work.

This is an exciting mode of working for us, though, to create new dramaturgical approaches to algorithmic work.

But I think these risks are in everything that is truly real time! It's the difference between producing something that is "fixed" or running in real-time. I think many people in creative coding are accustomed to these risks.

However, there are lots of times during the process where it's very difficult to know if it's working or not. We have to adjust a lot of different parameters that define the difficulty and/or game-mechanics. There is a lot of doubt that we have to work through.

Since we did a smaller work first--"Foxconn Frequency (no.2)"--we had a better understanding of what we were getting into with the third iteration of the series. I think we went into programming this third work knowing that the core experience was compelling. That starting point was helpful. The rest of it was trust and faith in our ideas, concept, execution, and our collaborators.

Derivative: My understanding is that the three performers are given increasingly complex musical scores to perform while being bombarded by noise, flickering lights etc. and that it is generally a HIGH STRESS scenario?

Remy Siu: Yes, it's a high stress scenario! It's definitely stressful. However, when I talk to the performers before and after the performance, I tell them that this work *accommodates* their failure. In fact, failure is required for the success of this work.

I think this is the compassionate part of the system. We wanted to make a space *for* failure and for people to witness, celebrate, and engage with failure. In the music world, especially classical piano, failure is generally not accepted. Failure is not tolerated in many parts of life.

I don't know if that relieves their stress, hahaha.

All the game modes in different containers. Each mode has separate [TABLE DAT]s for all the permutations of that mode.

The cueing system which is one python script being replicated by [REPLICATOR COMP]. The table used with the replicator contains all the parameters for the cue to be fed to the script.

Derivative: Can you explain how it works?! Both as the performance and how the audience experiences it and as the show is actually made with TouchDesigner.

Remy Siu: Let me try!

We have three stations on stage. Each station has a 88-key MIDI piano keyboard, two webcams, a speaker, a 3D-printer, and a computer monitor. These devices are plugged into the computer monitor USB hub and extended to the master computer tower.

All the projectors (one for each station and the main projector) are also plugged into the master computer.

One specced out PC Tower runs the whole show. We have two TouchDesigner processes: the main process and the 3D printer process.

All of the piano testing stuff is done in python, using TouchDesigner's local storage. Every time the performer presses a key, it runs a little python script that determines whether it is accurate based on the score the performer has been given.

For piano, we had to factor in different play styles. Some play styles are very staccato and detached, others are very legato, with the pianist holding the notes as they move their fingers. We had to make sure to support legato playing and also "thumb-under" movements.

We also have a custom cueing system where we can essentially compose the work in a [TABLE DAT]. There are a lot of nested tables in this work, with all the settings for each mode, the settings inside the mode, and all of those are called by our master cueing table.

We used the [REPLICATOR COMP] in our cueing system to duplicate a single python script that takes in a cue number and runs everything properly.

Once the piece begins, it is completely automated, moving forward only if certain conditions are met.

The performers receive instructions that the audience cannot see. Certain stats about the performer (how well they're doing, how much they have left to do, etc.) is shared knowledge between the performers and the audience.

Obviously, I don't have the room here to explain each mode and how they're programmed! Lots of [TRAIL CHOP]s and [CHOP EXECUTE DAT]s. A lot of python. And we use local storage *a lot* to hold all our parameters.

The [OP VIEWER TOP] is used to show the audience player performance over time. It's nice that the tool itself gets to show up in the show


Remy Siu: In some of the images, you can see that there are actually scores with Western music notation. These are what the performers see in general testing. As seen in the images, these scores are living in the system's 'memory,' to be called when needed.As anyone with programming experience might notice by looking at my TouchDesigner network and Python code, this process can get pretty messy. In many ways, the process of Foxconn Frequency (no.3) is informed by my experiences in devised theatre. In our rehearsals, I wanted to follow a promising trajectory if it suddenly appeared—for a split second—in front of us. This meant diving into the code and doing things quickly as opposed to cleanly.One can see this too, in the list/table of cues and various parameters. Some like, "FILLER," seem tacked on. That's because they were. It was probably not the best place to put this kind of parameter, but it was born out of the necessity for speed. Because oh my god, the performers look cold and agitated, we better get this thing on its feet in the next ten minutes.

Derivative: At one of the shows the three performers were stuck for 20 minutes unable to complete one of these tasks and apparently they were texting each other on-stage to see what to do to get out of it... How was that resolved? But also what did that feel like? When do you start panicking or hitting the ‘bail’ button?

Remy Siu: They started talking to each other during the performance! They also took a break at one point to collect themselves.

I don't give the performers many instructions on how to complete the work. I believe the only rule I give them is that they cannot leave their box to talk to one another. In rehearsal, they started shouting to one another.

I really like this aspect of teamwork and cooperation on stage, so we kept it in.

The section they were stuck in was called internally the "unison" mode. The three performers needed to play a passage in unison. The window for this unison was 0.8 seconds. We see unison in dance and music all the time, but this needed to be an exact, measured unison for them to succeed.

It felt pretty stressful backstage watching this occur. I think about hitting the "bail" button all the time, but I never do. I think the audience would feel cheated -- the performers too -- if I ever did anything like that.

Backstage, I generally never touch the computer after pressing GO. I don't want to intervene on the process unfolding on stage.

Derivative: An audience member at the Toronto performance mentioned that it’s very tense but that as the audience you also find yourself really excited by what’s going on… that it might feel very wrong but it’s as if you were watching a type of sport or even gladiator sport. Is that one of the intentions? 

Remy Siu: Definitely the tense part. We knew that it's tense watching these performers try to achieve the task.

We also see the performers competing with one another as pianists--I think this triggers something in the audience--musicians are not supposed to compete! Of course, the reality is, the concert pianist world is very competitive and cut-throat.

The system is designed to be subversive about this kind of competitive virtuosity.

Derivative: At what point in the design / development process was TouchDesigner first being used?  i.e. How was it used in the early stages?

Remy Siu: TouchDesigner was used in all aspects of the design and development process. We used TD to program all the game mechanics, the audio/visual feedback for the performers, and what the audience sees.

TouchDesigner was used throughout the whole project. Once we got the piano input and accuracy testing working, we started to flesh out lots of different minigames. This is where the real-time nature of TD was really great. We would adjust difficulty parameters on the fly to get everything just right.


Derivative: In each performance, 3D-printers are programmed to output a 'perfect cube' documenting the performers' competency over time.  What did these objects really look like? Were they at all similar throughout the 3 shows?

Remy Siu: These objects are cubes-- the recorded competency is the x-axis and time is the y-axis. We used to also use the z-axis, but because we don't know what the final output will be, it's difficult for structural integrity of the print if there is too much z-axis movement, and also, it made the output less clear.

They are printed differently every show, as every show is different.

Every time the printers start a new layer, it checks in with the main TouchDesigner process for the competency data, and adjusts the output accordingly. 

Derivative: Were there any surprises? “Aha moments” that you were not expecting?

Remy Siu: There were many surprises in process. A lot of the time, the performers found ways to break or cheat the system. Sometimes we "patched" these for balance, and sometimes we left them there, because the action of breaking or cheating the system was interesting.

Derivative: Did you learn anything new?

Remy Siu: Many many things. 3D printing in real-time. Lots of new best practices. Next level optimization. Things I didn't really want to learn, but had to--USB Power Management, Matrox wrangling, and a lot of other boring little bits that became really essential.  

Derivative: Has this project given you new ideas for future projects or ways of doing things?

Remy Siu: My practice has been focused on real-time game mechanics in the performing arts for the last four years. Every project, we learn something new, and we take this knowledge into the next project.

I definitely want to make use of the real-time 3D printing code again in some future project!

Also, with these real-time game works, there is an affect of "live-ness," of real "presence," that we always want to achieve. It's usually about the friction between performer and system. I think we better understand how to get to that place faster, so that we can build on it further.

Derivative: What has the response been – well, you are nominated for an award so I imagine the response has been very positive!

Remy Siu: It really depends on the age group that comes to see the work.

We take a kind of "Miyamoto" approach in regards to our audience. We try to demonstrate the game mechanic through experience and not through some kind of didactic. It's boring to have to get up and explain how the mechanics work before the performance. It's better that the audience comes to understand the mechanics through causality, drama, etc.

Sometimes, an older generation may have a hard time reading the game-mechanics. Usually, the younger generation gets it much faster. They watch Twitch, they play video-games, they code maybe--all of these aspects help with the understanding of the mechanics.

I feel, in the arts, we should be able to read algorithm the same way we read other kinds of formal elements. I think as more artists work in creative coding, or do things like this, audiences will come around, regardless of age.

After all, we interact with algorithms on the daily--we should strive to understand and recognize them, so that they don't take advantage of us.

Derivative: Is there anything else you would like to talk about or highlight?

Remy Siu: I work with a lot of different collaborators, and I want to make a shout out to them! A frequent collaborator is Paul Paroczai, who does a lot of the audio programming and sound for these works.

We're working on several new works, one which will premiere in Toronto in July 2019 at the Theatre Centre. And another, an algorithmic theatre sci-fi BIKE CHASE, which will probably premiere in a couple of years.

My website is always updated with this stuff! Thanks for reading and hope to see you in person at one of the shows, one day!

Remy Siu
CREATORS Hong Kong Exile (Natalie Tin Yin Gan, Milton Lim, Remy Siu) with Vicky Chow, Paul Paroczai, and Matt Poon
PERFORMERS Natalie Tin Yin Gan, Vicky Chow, Matt Poon
TEXT Xu Lizhi (1990-2014)
SUPPORTED BY Canada Council for the Arts, Banff Centre for Arts and Creativity
COMMISSIONED BY PuSh International Performing Arts Festival