Introduction
Integrating real-time sensor data (in this case movement) into TouchDesigner can significantly enhance interactive media projects.
Utilizing a microcontroller like the ESP32S3, particularly the LilyGO T-Display S3 developement board, in conjunction with the MPU6050 gyroscope and accelerometer sensor, we can develop a motion tracking system for capturing and transmitting data in real time.
This, enables the creation of dynamic visualizations, responsive environments and interactive art installations.
I built and open-sourced a mocap system that is able to calibrate on boot, collect and filter the data using expotential smoothing and then transmit the data in real-time to Touchdesigner.
The system is modular and you can iimplement more sensors using a multiplex module.
Key Steps for Integration
Components Needed
- LilyGO T-Display S3 (ESP32-S3 Development Board)
- MPU6050 Gyroscope and Accelerometer Sensor
- Push Button with LED
- Haptic Feedback Module (Vibration Motor)
- 1200mAh LiPo Battery
Assembly
- Connect the MPU6050 to the ESP32 using I2C communication (SDA to GPIO17, SCL to GPIO18).
- Attach the vibration motor to GPIO10 for haptic feedback.
- Integrate the push button and LED to GPIO3 and GPIO2, respectively, for user interaction.
Software Setup
- Setup the Arduino IDE (if you use the LilyGo T-Display S3) and install necessary libraries for MPU6050 and WebSocket communication.
- Clone the repository from my github page and upload the .ino code from the project's repository to the microcontroller. There are two versions, with or without User Interface implemetation (related to the TFT display).
Node.js Server:
- Go into the server folder from the repository and install the packages (using `npm install`), then start the WebSocket server (using `node server.js`) to facilitate data transmission between the ESP32 and TouchDesigner.
TouchDesigner Configuration:
- Utilize WebSocket DAT and CHOP operators to receive and process incoming sensor data.
- Map the motion data to visual elements for real-time interaction.
For a step-by-step guide and technical details on building this mocap system, including detailed code examples and hardware schematics, please visit the full article from the link below.
