Experimental:OAK-D

(available in experimental TouchDesigner 2023.10000, and by request in 2022.Official)

OAK-D cameras offer a range of high-resolution cameras with depth vision and on-chip machine learning.

OAK cameras do computer vision tasks on-device and send the processed images or data to TouchDesigner on both Windows and macOS. For example, an OAK camera can compute human skeleton landmarks for a live RGB image and send both the landmarks and image to TouchDesigner. OAK is short for the OpenCV AI Kit, which originated through online crowd-funding. Luxonis has expanded the line of OAK hardware and also released several open-source libraries such as DepthAI that run on the OAK.

Three operator types work together to implement the OAK functionality: The OAK Device CHOP is the main interface to the camera devices. The OAK Select CHOP extracts channel and dictionary data from the camera results of a OAK Device CHOP. The OAK Select TOP extracts processed images from the camera. Each DepthAI model running on each camera will have one OAK Device CHOP plus one or more of the OAK Select OPs.

The aim of integrating TouchDesigner with OAK cameras is to facilitate novel interactions with the rest of the TouchDesigner environment while providing low-latency, high throughput performance with the OAK hardware. If you think a pure Python example that doesn't involve TouchDesigner is running faster than one which does, please let Derivative know. For example, Derivative aims for the frames-per-second of an OAK's RGB camera received in TouchDesigner to be at least as fast as what you would receive in pure Python with DepthAI and OpenCV.

Note: A Time-of-Flight version of the camera with depth-model support is in development as of September 2023.

Requirements

The OAK operators are currently only supported for Windows.

Installation

No installation steps are necessary. TouchDesigner's installation includes the depthai Python module as a third-party package. For simplicity, we ask that you not install depthai to any custom Python environments, even if they're not being used with TouchDesigner.

Usage

The example file is DepthaiTestSuite.toe. Instructions and tips are inside the file. I future builds, the file will be located in C:/Program Files/Derivative/TouchDesigner.2023.xxxxx/Samples or by request.

Troubleshooting

If you encounter issues with the depthai examples, consider following these steps to help Derivative fix the issue.

Confirm that the depthai Python module being used is from TouchDesigner's site-packages folder. Run this script in a DAT
```
import depthai as dai
print('depthai version:', dai.__version__)
print('depthai location:', dai.__file__)
```
The printed location should be inside TouchDesigner's site-packages folder, not a custom Python environment.
Isolate the issue to the minimal case. Delete any TouchDesigner nodes which aren't necessary to reproduce the issue.
Set the following system environment variables. Set DEPTHAI_LEVEL to debug. Also set TOUCH_TEXT_CONSOLE to 1.
Open the TouchDesigner project with the issue and then press alt-c to open the console. Follow whichever steps lead to the issue while observing the output in the console. Share detailed instructions of the necessary steps and the console output.

Power over Ethernet

Some OAK cameras connect via USB and others connect via Power over Ethernet (PoE). PoE can be used over longer distances than USB cables, but latency will be higher. At short distances, USB3 would still be faster than PoE. If you are seeing issues with PoE examples, try lowering the amount of information being exchanged over the network. For example, you could decrease the FPS of cameras, use a smaller RGB/mono resolution, or use an ISP scale other than (1,1).

A getting started and installation guide for the poe version is located on the Luxonis Website's Helppages here: https://docs.luxonis.com/projects/hardware/en/latest/pages/guides/getting-started-with-poe/

Privacy policy

Follow us