RF-DETR for TouchDesigner For Beginners & Intermediate

A TouchDesigner TOX wrapping Roboflow's RF-DETR transformer model with a GPU-composited render path and a zero-Python-thread architecture. Ten model variants (4 detection + 6 segmentation, all COCO-pretrained) swap from a dropdown. Output is a composite TOP plus a transparent overlay TOP — wire either or both into your network.

What's in the download

RFDETR.tox - self-contained TOX (12 KB, all Python embedded) README.md - usage docs INSTALL.md - install / troubleshooting per platform requirements.txt - pip list — used by the in-TOX installer python/ - source files - (extension, callbacks, shader) screenshots/ - 13 sample outputs across every viz mode

Drop the TOX into any project, click Setup → Install Dependencies, click Setup → Predownload All Models if you want to skip future weight downloads, then pick a variant from the Model dropdown.

Performance — apples to apples

Measured on M-series Apple Silicon (MPS), 720p source frame, all visualisations on (mask + bbox + labels). Each cell is an EWMA over 30 warm cooks. Display FPS = throughput of the full pipeline cooking.

ModelResolutionInferenceMask paint (CPU)Overlay paint (CPU)GLSL compositeDisplay FPSSegmentation Medium43225.3 ms0.8 ms0.3 ms≈ 0.5 ms30Segmentation Nano31220.7 ms0.8 ms0.4 ms≈ 0.5 ms32Segmentation Small38420.1 ms1.1 ms0.4 ms≈ 0.5 ms31Detection Nano38419.7 ms0.2 ms0.4 ms≈ 0.5 ms35

If you have NVIDIA, all of this gets faster — torch on CUDA is roughly 2× MPS for the same model at the same resolution. The bottleneck is inference; the surrounding pipeline is fast enough that adding more GPU power moves the needle directly.

How it compares to YOLO / TDYolo

Source for these numbers: Roboflow RF-DETR post and Ultralytics YOLO11 docs.

On the TouchDesigner side specifically, vs typical community YOLO packs ("TDYolo" and variants):

• No background Python threads. Every TD object read happens on the cook thread. The "TouchDesigner objects cannot be referenced from separate threads" warning is impossible. grep -nE 'threading\.Thread\(|import threading' python/extension.py returns nothing.

• GPU compositor. Most TD wrappers I've seen build the visualisation with numpy / cv2 inside the Script TOP onCook. With a segmentation model that means full-res mask alpha blends every frame — that's exactly the 30-80 ms / frame I was hitting before pulling the composite into a GLSL TOP.

• Subprocess install + weight pre-download. Pip runs as a real OS subprocess; TD stays responsive. A second pulse instantiates every variant in a separate Python process to trigger their HuggingFace downloads up-front, so the dropdown becomes a no-network operation afterwards.

• MPS supported**, segmentation as a first-class output, and doesn't hang TD when you change the model variant, this is the most honest pick right now.

Optimal use scenarios

Realtime interactive installations. Stable 30 FPS at 720p with a mid-tier Mac means you can drive lighting / sound / generative output off detected objects without dropping frames or burning a separate machine.

Generative art with mask isolation. Wire out_overlay into a Composite TOP, drop it on top of arbitrary GLSL, get clean isolated subjects as a transparency mask. The "Hide Source (Overlay on Black)" toggle turns the main output into the same shape on a black canvas for direct alpha use.

Data-driven control surfaces. The CHOP output emits class_id, confidence, x1/y1/x2/y2, area, and frame per detection. Feed into a Select CHOP filtered by class id, then to anything — sliders, LFO frequencies, particle counts, parameter expressions.

Custom model deployment. The Custom Checkpoint Path parameter loads any .pt produced by rfdetr.train(…) — train on your own dataset, drop the file in, run.

Multi-camera / multi-instance. Drop multiple copies of the TOX in a project, each with its own input TOP. Each instance loads its own model into MPS / CUDA memory — RAM permitting (the Nano variants need ~ 200 MB each on GPU; XL variants need ~ 1.2 GB).

Mac + Windows live setup. macOS Apple Silicon and Windows + NVIDIA both supported in the same TOX. The installer detects what's there and pulls only what's missing. Pre-show: predownload every variant once on the target machine, then the dropdown is a no-op.

What it's NOT ideal for. Sub-10ms latency requirements (use ONNX + TensorRT). Server-side batch processing (use rfdetr directly). Very small object detection at distance (RF-DETR Large is better than Nano here, but no segmentation model handles 16-px-diameter faces well).

Roadmap / things I might add next

• ONNX export path in the Setup page — exports the current model variant for downstream CoreML / TensorRT.

• Multi-source batching — one model instance, multiple Script TOPs feeding it, frame-time batch inference.

• Tracker integration — supervision's ByteTrack for stable per-instance IDs across frames.

• Pose estimation variant — RF-DETR has experimental pose variants; would need a new visualiser.

If any of these would be useful for what you're building, drop a comment.

Support / continued work

Direct support keeps the lights on for builds like this:

• Patreon: patreon.com/c/aritd — contributor tier unlocks downloads.

• Buy me a coffee: buymeacoffee.com/aristides.lab

• Feedback: comment on this post, or message me on Instagram

License & credits

• TOX, Python, shader sources — MIT.

• RF-DETR — Apache 2.0, Roboflow Inc. The XL and 2XL variants require a separate Roboflow commercial license for commercial use.

• supervision — Apache 2.0, Roboflow Inc.

• COCO classes — 80-class subset.