Mediapipe AI + Nvidea Flex Solver ┃ Interractive Particle system

This is an Interactive Digital Energy Field modified according to the hand's motion which is tracked through AI integrated into Touchdesigner. 

I have designed a project that is integrated with MEDIAPIPE Hand tracking to control Nvidea Flex Solver generative particle system + sliders for additional parameter adjustments. It's not too complicated to understand the controls here, there are a couple of highlights that create this magical interactiveness.
 

To get started:

1. Download the Mediapipe AI plugin (see links below).
2. Once downloaded, drop the project.tox file you receive into the Mediapipe plugin folder.
   
   ( BTW big thanks to Torin Blankensmith for the original GPU Accelerated MediaPipe Plugin for TouchDesigner ).

• Startup: The video feed activates automatically, and MediaPipe AI begins analyzing your hand movements.
• Simulation Launch: Press the start button, and a swarm of particles fills the space.
  
   

Single-Hand Gestures:

• Gravity at Your Fingertips: Point your index finger upwards, and watch the particles rise as if pulled by an invisible force. Intuitive Control - The overall gravity direction of the particles responds to your hand's orientation.

• Depth Control: Move your hand closer to the camera, and the particles condense. Move it further away, and they scatter chaotically.

The main SOP object is the invisible shape force that acts like a magnetic center for the particles. with double comp feedback I have made this shape deformable, so eventually, we get a whole bunch of abstract magnetic objects created according to the operation mode of these two comps :

note: it is made switchable like this so there are options to control both shape creators with sliders and also with hands
Bonus: You can even switch between pre-defined SOP shapes.

 

Dual-Handed Interaction:

• Right-Hand Pinch: Imagine trapping the particles! A pinch gesture with your right-hand binds the particles to their initial position.

• Left-Hand Pinch: you control the first comp shape / scroll through blend mode operations that take affect in feedback loop. Experiment with different shapes to see how it affects their movement and behavior

All the external sliders that are connected to the main solver parameters are pretty self-explanatory here:

Note: This project currently requires the 2023 version of Touchdesigner.
Important Note: you have to first download the Mediapipe AI plugin from the link below and drop the project.tox file into the Mediapipe folder to be able to launch it properly.

This interactive installation is yours to explore and experiment with. Conduct the energy field with your hands, sculpt its form, and witness the captivating dance of particles. Feel free to use it in any way that sparks your creativity! I'd love to hear your feedback and see the amazing results you create.
---May the force be with you---