Close

Privacy policy

Follow us

https://www.instagram.com/touchdesigner/
https://www.facebook.com/TouchDesigner
https://twitter.com/1null1
https://www.youtube.com/TouchDesignerOfficial
All
User Guide
Forum
Tutorials
Events & Workshops
Features
Community
Showcase
Resources
  1. Home
  2. Experimental:OAK-D

    Experimental:OAK-D

    (available in a future build of TouchDesigner)

    OAK cameras can do computer vision tasks on-device and send the processed images or data to TouchDesigner. For example, an OAK camera can compute human skeleton landmarks for a live RGB image and send both the landmarks and image to TouchDesigner. OAK is short for the OpenCV AI Kit, which originated through online crowd-funding. Luxonis has expanded the line of OAK hardware and also released several open-source libraries such as DepthAI that run on the OAK.

    Three operator types work together to implement the OAK functionality: The OAK Device CHOP is the main interface to the camera devices. The OAK Select CHOP extracts channel and dictionary data from the camera results of a OAK Device CHOP. The OAK Select TOP extracts processed images from the camera. Each DepthAI model running on each camera will have one OAK Device CHOP plus one or more of the OAK Select OPs.

    The aim of integrating TouchDesigner with OAK cameras is to facilitate novel interactions with the rest of the TouchDesigner environment while providing low-latency, high throughput performance with the OAK hardware. If you think a pure Python example that doesn't involve TouchDesigner is running faster than one which does, please let Derivative know. For example, Derivative aims for the frames-per-second of an OAK's RGB camera received in TouchDesigner to be at least as fast as what you would receive in pure Python with DepthAI and OpenCV.

    Requirements

    The OAK operators are currently only supported for Windows.

    Installation

    No installation steps are necessary. TouchDesigner's installation includes the depthai Python module as a third-party package. For simplicity, we ask that you not install depthai to any custom Python environments, even if they're not being used with TouchDesigner.

    Usage

    The example file is DepthaiTestSuite.toe. Instructions and tips are inside the file.

    Troubleshooting

    If you encounter issues with the depthai examples, consider following these steps to help Derivative fix the issue.

    1. Confirm that the depthai Python module being used is from TouchDesigner's site-packages folder. Run this script in a DAT 
      import depthai as dai
print('depthai version:', dai.__version__)
print('depthai location:', dai.__file__)
      The printed location should be inside TouchDesigner's site-packages folder, not a custom Python environment.
    2. Isolate the issue to the minimal case. Delete any TouchDesigner nodes which aren't necessary to reproduce the issue.
    3. Set the following system environment variables. Set DEPTHAI_LEVEL to debug. Also set TOUCH_TEXT_CONSOLE to 1.
    4. Open the TouchDesigner project with the issue and then press alt-c to open the console. Follow whichever steps lead to the issue while observing the output in the console. Share detailed instructions of the necessary steps and the console output.

    Power over Ethernet

    Some OAK cameras connect via USB and others connect via Power over Ethernet (PoE). PoE can be advantageous for long distances, but latency will be increased. At short distances, USB3 would still be faster than PoE.

    See Also