Close

Privacy policy

Follow us

https://www.instagram.com/touchdesigner/
https://www.facebook.com/TouchDesigner
https://twitter.com/1null1
https://www.youtube.com/TouchDesignerOfficial
All
User Guide
Forum
Tutorials
Events & Workshops
Features
Community
Showcase
Resources
  1. Home
  2. OAK-D

    OAK-D

    OAK-D cameras from Luxonis offer a range of high-resolution cameras with depth vision and on-chip machine learning.

    OAK cameras do computer vision tasks on-device and send the processed images or data to TouchDesigner on both Windows and macOS. For example, an OAK camera can compute human skeleton landmarks for a live RGB image and send both the landmarks and image to TouchDesigner. OAK is short for the OpenCV AI Kit, which originated through online crowd-funding. Luxonis has expanded the line of OAK hardware and also released several open-source libraries such as DepthAI that run on the OAK.

    Three operator types work together to implement the OAK functionality: The OAK Device CHOP is the main interface to the camera devices. The OAK Select CHOP extracts channel and dictionary data from the camera results of a OAK Device CHOP. The OAK Select TOP extracts processed images from the camera. Each DepthAI model running on each camera will have one OAK Device CHOP plus one or more of the OAK Select OPs.

    The aim of integrating TouchDesigner with OAK cameras is to facilitate novel interactions with the rest of the TouchDesigner environment while providing low-latency, high throughput performance with the OAK hardware, and offloading some compute to an external device.

    Note: If you find a pure Python DepthAI / OpenCV example that doesn't involve TouchDesigner is running faster than one which does, please let Derivative know. For example, Derivative aims for the frames-per-second of an OAK's RGB camera received in TouchDesigner to be at least as fast as what you would receive in pure Python with DepthAI and OpenCV.

    The OAK-D is an open source hardware design.

    Note: A Time-of-Flight version of the camera with depth-model support is in development as of Summer 2024.

    Installationedit

    You must select OAK-D when installing TouchDesigner. No other installation steps are necessary. TouchDesigner's installation includes the depthai Python module as a third-party package. For simplicity, we ask that you not install depthai to any custom Python environments, even if they're not being used with TouchDesigner.

    Models will download once the first time they are used.

    Usageedit

    The example file is OAKExamples.toe. Instructions and tips are inside the file. The file is located in C:/Program Files/Derivative/TouchDesigner.2023.xxxxx/Samples/OAK.

    NOTE: To use this file, please copy it to a folder outside the regular program location like your user documents or desktop.

    Models by default will be downloaded to %USER%/.cache/blobconverter or %USER%/.cache/TDFileDownloader depending of the source of the model.

    Troubleshootingedit

    If you encounter issues with the depthai examples, consider following these steps to help Derivative fix the issue.

    1. Confirm that the depthai Python module being used is from TouchDesigner's site-packages folder. Run this script in a DAT 
      import depthai as dai
print('depthai version:', dai.__version__)
print('depthai location:', dai.__file__)
      The printed location should be inside TouchDesigner's site-packages folder, not a custom Python environment.
    2. Isolate the issue to the minimal case. Delete any TouchDesigner nodes which aren't necessary to reproduce the issue.
    3. Set the following system environment variables. Set DEPTHAI_LEVEL to debug. Also set TOUCH_TEXT_CONSOLE to 1.
    4. Open the TouchDesigner project with the issue and then press alt-c to open the console. Follow whichever steps lead to the issue while observing the output in the console. Share detailed instructions of the necessary steps and the console output.

    Power over Ethernetedit

    Some OAK cameras connect via USB and others connect via Power over Ethernet (PoE). PoE can be used over longer distances than USB cables, but latency will be higher. At short distances, USB3 would still be faster than PoE. If you are seeing issues with PoE examples, try lowering the amount of information being exchanged over the network. For example, you could decrease the FPS of cameras, use a smaller RGB/mono resolution, or use an ISP scale other than (1,1).

    A getting started and installation guide for the poe version is located on the Luxonis Website's Helppages here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/getting-started-with-poe/

    See Alsoedit