How AI is Transforming Interactive Experiences with TouchDesigner
I'm Davide Carbone, a creative engineer at LivingSparks Studio .
Our goal is to partner with artists to bring their creative visions to life through innovative technology. We specialize in creating installations and visual performances, primarily utilizing TouchDesigner.
I started using TouchDesigner a few years ago, mainly for controlling systems in mechanical installations. Over time, I’ve begun to incorporate more elements into my projects, like shaders, lasers, and lately, computer vision.
In this blog post, I'm going to share my experience with using OpenCV and ONNX to integrate computer vision and machine learning models into TouchDesigner projects.
Using OpenCV with TouchDesigner
I recently came across the Script TOP in TouchDesigner, which enables the generation of TOP images through Python scripting. One particularly useful feature it offers is copyNumpyArray()
. This function allows you to read in python the content of a TOP providing it with a NumPy array containing pixel data.
OpenCV is a powerful library for computer vision, and it can be used for a wide variety of tasks, such as image processing, object detection, and facial recognition.
This discovery allowed me to apply my knowledge of OpenCV directly within TouchDesigner. Processing live video streams and experimenting with various computer vision techniques, all while staying within the TouchDesigner environment.
This integration has not only improved my workflow but also broadened the possibilities for creating interactive installations and visual performances.
Getting Everything to Work Together
To achieve this, I had to conduct extensive research in different areas. The main challenge was to merge these different “blocks of knowledge” to make the system function precisely and that the system works in real-time.
One of the initial challenges was installing the correct libraries. This step was crucial to ensure compatibility and smooth operation within the TouchDesigner environment. For managing Python environments and third-party libraries in TouchDesigner I recommend this fantastic blog post: Anaconda - Managing Python Environments and 3rd Party Libraries in TouchDesigner.
Tip: Adding Your Scripts
If you want to add your custom scripts to the system, include this line in your code:
sys.path.append(project.folder+'/scripts')
Create a folder named "scripts" inside your main project folder. This way, the Python interpreter will include the "/scripts" directory in its search path, allowing you to easily import and use your scripts.
Why I Chose ONNX
After some experimentation, I initially tried integrating PyTorch with TouchDesigner. Unfortunately, I encountered issues with NumPy version compatibility, preventing PyTorch from functioning correctly within the TouchDesigner environment. PyTorch is typically my go-to framework for AI model integration due to its user-friendly nature. And generally it’s not the most performing one.
To overcome that, I turned to ONNX, which stands for Open Neural Network Exchange. ONNX is an open standard for representing machine learning models, allowing them to be transferred between different frameworks platforms and OS, without losing information.
The advantages of using ONNX are:
-
Compatibility: in general ONNX models work with multiple deep learning frameworks like PyTorch, TensorFlow, and others. This made it easier to integrate models without worrying about specific library dependencies.
-
Efficiency: ONNX is optimized for performance, which means faster execution and real-time processing—crucial for interactive and real-time applications in TouchDesigner. ( they can run on GPU)
Trying Other Integrations
To be thorough, I also tested other integration methods like using OpenCV's cv2.dnn.readNetFromONNX
However, the performance wasn't as good as using ONNX directly.
One of the biggest challenges is converting models. There are tons of PyTorch models available online, but not many are already converted to ONNX—you usually have to do that part yourself. I found this article really helpful for getting started with conversions: Converting a PyTorch Model to ONNX Format.
And it's not just about image-to-image or image-to-features models. ONNX has a huge range of models covering everything in the AI world, like image-to-text, text-to-image, and even text generation models like GPT-2 and others.
The Game Changer
I initially used OpenCV's drawing functions within a Script TOP in TouchDesigner for my visualizations. While this worked, I sought to optimize performance and gain finer control over graphical elements. The solution? Leveraging TouchDesigner's native capabilities.
By exporting keypotins and features data to a DAT table and utilizing native instancing, I dramatically improved rendering efficiency and workflow. Complex graphics became smoother, more responsive, and easier to manage in Touchdesigner.
Right now I've successfully integrated YOLO and Ultraface models into this optimized pipeline. Using a 3080Ti notebook GPU, I achieved the following performance:
-
YOLO: 43 FPS without segmentation, 33 FPS with segmentation
-
Ultraface: Approximately 70 FPS
My next goal is to replace traditional computer vision models with LLM and visual recognition models for real-time text-to-text and image-to-text captioning.
Conclusion
The challenges encountered—ranging from library compatibility issues to the complexities of model conversion—have been valuable learning experiences for me.
As we continue to explore and push the boundaries of these technologies, there are exciting opportunities ahead for collaboration and experimentation in this field.
Next Steps
Model Integration: I plan to incorporate Phi-3 Vision into the existing model stack.
System Testing: I'll evaluate the system's performance within the ENGINE environment to optimize Touchdesigner's multi-process capabilities.
Let's Create Something Together
I'm eager to experiment with these models on various subjects. If you're a filmmaker or simply passionate about visual storytelling, let's connect! I'm open to collaborating on video projects