Deepfake Salvador Dalí Interacts with Museum Visitors, Takes Selfies!

Dalí Lives is unique in that it offers a new way to delight and inspire museum visitors and is a remarkable example of art meeting artificial intelligence and showcasing the possibilities created by this sometimes controversial technology.

We spoke to creative technologist Shan Jin of the San Francisco-based agency Goodby Silverstein & Partners (GS&P) who partnered with the museum to create the immersive exhibit. Shan Jin works at GS&P Labs, an internal innovation department at GS&P that experiments with emerging technology and collaborates with other creative departments to develop prototypes and fun experiences. Shan managed the Dalí Lives project and was responsible for the TouchDesigner development aspects of the implementation.

Derivative: Shan can you describe the Dalí Lives experience for us?

Shan Jin: When you arrive at the museum, you will find what looks like a free-standing door, but it’s actually a person-height vertically oriented screen. This is our first screen - the “welcome screen”. Visitors will first see a man in the distance, walking back and forth, holding a cane, reading a book, fixing his moustache, or painting. If they choose to press the button and interact with the screen, a man gets close to the visitor stepping into the light and it becomes apparent that it’s Dalí himself: alive and life-sized. Dalí will then greet visitors and comment on current events, the day of the week, and the weather, for example. 

Screen 2 sits on the second floor where Dalí talks about his artwork, the motivations behind his work, his childhood or his love for Gala. Dalí would just be sitting there, or reading a newspaper and suddenly begins to talk whenever the button is pressed. A new feature that we are working on right now is that we’ll render new videos everyday with the front page of the Tampa Bay Times, so you’ll always see Dalí reading the latest newspaper. We’re hoping to roll out that feature in the next a couple of weeks. Each visitor’s interaction with Dalí might be a little different from the next and not everyone will necessarily get the same experience.  We added some dynamic content and various different versions, so it feels like Dalí always has something different to say. 

Image Description: The split panel on left is the root path, the red node “Variable1” is a master operator(container) for the replicator, and all the yellow containers are auto generated (in each container Dalí does different things, talk about weather, temperature, days of week, time of day etc etc.). The right split panel is inside “Variable1”, the red node “Condition1” is also a container for replicator, because under each variable there are different conditions (like under temperature there are “warm”, “mild”, “cold”). So for each container generated the condition also gets generated automatically. It reads the folder structure and does the work based on that. 

Image Description: Under container “Condition1” there are operators who does the real work, like a container for playing different videos for idle states, switch between idle and contents.

D: How was TouchDesigner used to bring Dalí to life?

SJ: For screen 1 and 2, TouchDesigner holds together all the face-swapped assets from motion. It reads the button input from serial and triggers the content. I used the Replicator so that it’s more organized to play the 125 videos we have. When the videos are placed in the right folder structure, all the containers are generated automatically. It also makes it really easy if we need to update a new version of the contents - given the right folder path TouchDesigner would just pick up whatever video file is inside that folder. I actually have replicator inside replicator because we label the contents as different variables - conditions - idle/content, and it’s very satisfying to see that every time those containers are generated recursively. Components also come in handy for reusing the module across different screens.

Image Description:This is the network inside corner_pinning, it loads the data from csv, check if the current frame is larger than the start frame, and loads the 4 position data based on row number.

SJ: There’s also Python script inside TouchDesigner to help with the dynamic contents. For example we’re pulling real time weather API so that on a rainy day you’ll see Dalí walking with an umbrella. Or on a Thursday Dalí would tell you the museum opens late that day. A timer will trigger our logo animation every 30 seconds to prompt users to push the button, because we found out that people are really cautious to touch anything inside the museum! 

Screen 3 sits in gift shop where Dalí takes a selfie with visitors - and they can text Dalí to get the image! For this one there’s a lot of close collaboration with the motion department. We have 7 different versions for selfies, and each of the components have their own local timeline. When Dalí holds up the phone we use corner-pinning to map the webcam feed to the phone. The corner-pinning data is exported from After Effects, I set the local timeline’s frame rate to match the frame rate of the video and lock the video playback to the timeline.

D: Which part of this process had you not done before?

SJ: The sheer volume of this project and the fact it’s a permanent installation in a museum is something I’ve never done before. My past experience focused more on events. Also managing the project and developing at the same time is a really big challenge for me, that I need to talk to ten different people every day and at the same time try to focus on software development. It’s a tricky balance in what I tried to delegate to other people as opposed to what I wanted to take on by myself.Time managing wise, towards the end of the project there was a lot more fine-tuning on the design side than I expected, it was hard to strike a balance between creative/engineering. Shall we keep making changes? Or shall we stop the development now and focus on testing? I think it’s always something to think about on project management for a tech lead.

Image Description: This is a sceenshot from selfie screen. The component has its own local timeline and the red nodes here are all Logic CHOPs trying to trigger different events based on the timeline (events like when the LED turns on, when to corner pin the camera feed onto screen, when to upload the image and get a code from the server).