In these days of COVID19, a significant part of our daily lives has switched into digital environments and endless video conference meetings.
I run an Art and Technology program at the University, and after having to shift our classes to online platforms for the rest of the semester, some students asked me to find ways to help them feel less isolated. Since we are doing our classes through Zoom now, I figured it would be cool to give them a tool to design their own dynamic camera backgrounds and portray themselves through Zoom in creative ways. Also using the iPhone camera allows us to stream higher image quality.
I wanted to share with you in this post, a relatively quick setup, that will allow you to capture your iPhone screen and stream it live to Zoom through TD.
- Spout, we will use SpoutCam. It only runs on Windows ( **for Mac Users there is a workaround described below).
- NDI HX Capture IOS app
- Install NDI tools
- TD educational or commercial license. In Jarret's latest post, he mentions that in next week's build, the NDI tools will also be available for non-commercial builds. As for now you'll need commercial.
Well, you will use the NDI HX app to capture your iPhone screen and stream it into TouchDesigner. In TD, you'll be able to process this footage and create multiple real-time effects; including chroma-keying the background and replacing it with your own real-time graphics.
This footage comes into TD using the "NDI In" TOP.
For Background Removal, you can use the iPhone camera app. Look for the "Portrait" lighting effects, and choose either:
- Stage Light Mono
- High Key light Mono
All of these iPhone camera features will remove the background and replace it with either black or white. Once you have this footage in TD, you can apply different TOP effects to clear the background. I added some examples in the TD file I'm sharing, but settings will depend on your facial features, clothing, lighting, etc
Once you have created your new background, you will send it to the "Syphon Spout Out" TOP operator.
Once Spout is set up correctly and your footage connected to the Spout OUT Top, go to Zoom.
In the Zoom interface, once you join or start a meeting, right-click next to the video camera and an option to use "spoutCam" will show pop up. Choose this camera.
Overall recap of the workflow:
Stream from your phone into TD
Image processing and real-time graphics in TD
Spout it out to Zoom.
I have only tried this with Zoom, but my guess is that spoutCam will work on other video streaming software (i.e. Skype) since the software detects it as another camera input.
You could also use "NDI Virtual Input" instead of Spout to stream to other social media platforms. Jarret explains this in his latest post.
Spout is not supported on Mac, you will use Syphon instead. You can download a Syphon simple server here (Run the client server) . Syphon does not seem to have a virtual camera feature at the moment so I used CamTwitst in order to share the TD texture with Zoom. Once Camtwist is installed you can choose the Syphon option. If you have issues setting up CamTwist you can follow detailed instructions in this post.
Camtwist will show up on Zoom's video camera options, similar to SpoutCam. In CamTwist preferences, you can change the frameRate and resolution of the live stream to match TD's.
Here is the link to the repo with a TD file:
I hope you enjoy this, and that it serves a mild distraction while in lockdown.
If you end up using it in any virtual meeting, please share your screenshots!