The WebRTC COMP is a component that can be used to initiate real-time communications (RTC) between TouchDesigner instances and TouchDesigner or WebRTC compatible devices such as most Web browser capable devices, including some IoT devices.
Before getting started with WebRTC, it is highly recommended to give a read to the introduction available on MDN Web Docs.
Reading this doc will help you get familiar with some of the vocabulary used in our WebRTC related docs.
There is a large consortium of actors working around WebRTC. With that in mind, TouchDesigner will attempts to not reinvent it in a way that would make it too different from what is already heavily documented. We have wrapped some of the features and flow available in the WebRTC API (native in modern web browsers) as well as the Media Capture and Streams API.
What it means for WebRTC in TouchDesigner: for most of the methods, variable names, callbacks... etc used, it should match the WebRTC API and / or the Media Capture and Streams API.
onNegotiationNeeded method in the webRTC COMP extension, is an event callback firing when one end of an RTCPeerConnection is requiring re-negotiation.
If you go to MDN, you will easily find more details about
onNegotiationNeeded with just a copy / paste of the method name.
The WebRTC API is detailed here: https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API
The MediaStream API is detailed here: https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API
- Because the Video Stream In / Out TOPs and Audio Stream In / Out CHOPs (some of the underlying operators of this component) are relying on Libwebrtc's own encoder / decoder and the library does not support hardware encoding / decoding, passing high resolution video can be resource intensive and users should be cautious. We are working on a solution to improve performances.
Good to know
Signaling is part of the WebRTC discovery and negotiation process.
While we do follow the recommended flows and messaging styles available at MDN and / or Google, including "Perfect Negotiation", there are a few nuances in our signaling process that does not make the WebRTC COMP plug'n'play with other Signaling APIs.
While we plan to add support for Unity and Unreal Engine versions of Signaling in the future (which are both different), advanced users will likely want to just take inspiration from the webRTC COMP and its WebRTCExt and adapt those to their own signaling needs.
i.e. You could run a pre-existing signaling server in NodeJS rather than our signalingServer COMP, and use a protocol different than WebSocket for exchanging messages, such as XMPP.
On the other hand, advanced users could also adapt signaling servers outside of TouchDesigner while still following our own Signaling API.
More details about the Signaling API are available at the following page.
A STUN (Session Traversal Utilities for NAT) is used to discover your public IP address and determine whether or not your NAT is open for this protocol and the required ports.
You can specify a STUN Server in the WebRTC COMP parameters dialog. Note: The default is a public STUN provided by Google. It is highly advised not to use this server in a production context.
A TURN (Traversal Using Relays around NAT) server could help in achieving a reliable connection between two or more peers and bypass NAT restrictions.
You can specify one TURN Server URL and Username / password in the WebRTC COMP parameters dialog. Additional TURN Servers can be added to the WebRTC DAT in the webRTC COMP.
Encryption is a mandatory feature of WebRTC. All components used in the WebRTC context are exchanging encrypted data.
Note that if you are concerned about security, a user should at the very least use a secured WebSocket (wss) connection at the signaling server level as well.
That being said, there are security considerations around WebRTC and its underlying technologies. Advanced users should read the following draft, especially when using WebRTC on a public network: https://datatracker.ietf.org/doc/html/rfc8826
Shortcomings and other considerations
Due to the complexity and the variety of applications around WebRTC, it is highly likely that this component could not be used (or at least, not fully) in a production context and environment.
One of the main concerns here would be scalability.
In most cases, you will want to rely on what we often refer to as SFU (Selective Forwarding Unit) or MCU (Multipoint Conferencing Units) to handle large groups of sources. A few names to look into or consider are AntMedia, Jitsi Videobridge, Kurento, Janus.. etc
While we didn't test this specific use case, advanced users could consider running TouchDesigner as an SFU or MCU.
Another important point is that it is likely that you will encounter firewall restrictions and a closed or moderate NAT. Proper STUN (and optionally TURN) server configurations should help with that. We recommend Coturn as a private TURN server.
To start a WebRTC connection between two TouchDesigner instances, either locally or on the local network, only the Palette components from the WebRTC folder are required.
In this example, we will consider two TouchDesigner processes running on the same machine, A and B.
In process A, drag n drop the signalingServer from the Palette → Change its Active par to ON.
In process A, drag n drop a signalingClient from the Palette → Change its Active par to ON. Change its Forward to Subscribers par to ON.
In process A, drag n drop a webRTC COMP from the Palette → Reference the signalingClient COMP to the webRTC COMP's signalingClient par.
In process B, drag n drop a signalingClient from the Palette → Change its Active par to ON. Change its Forward to Subscribers par to ON.
In process B, drag n drop a webRTC COMP from the Palette → Reference the signalingClient COMP to the webRTC COMP's signalingClient par.
Now if you look at the signalingServer, you’ll see 2 signalingClients showing up.
On each signalingClient, you will see the other signalingClient.
On each WebRTC COMP, you will see the other client (the client that represents a signalingClient which is not the one assigned to the WebRTC COMP you are looking at)
You just have to go in active mode to start a call.
In the WebRTC COMP viewer, after initializing a connection from a peer to another, click the manage tracks button.
Here, you can see all the current tracks, per connections, of the WebRTC COMP.
Using the ID dropdown menu at the bottom of the window, you can filter tracks per connection IDs. You can manually delete tracks using the RemoveTrack button in the table.
Finally, using the form at the bottom, you can add tracks of various types. Tip: You can drag an OP of a type matching the Type dropdown to the Source field.
When first initializing a connection from one WebRTC COMP to another, a default dataChannel is set. You can interact with the viewer of the WebRTC COMP with mouse clicks / moves and the position is being sent over a "Mouse" dataChannel. On the receiving end, a circle/pointer is being drawn and composited back into the image.
Additionally, you can go in the track manager (described in the previous section) and add a new dataChannel. Name the dataChannel
GenericCHOP (respect spelling, case sensitive) and select its type from the dropdown menu to be
dataChannel:CHOP. Last, create a CHOP of 5 channels of 1 sample each, non time-sliced. All those are mandatory as well and important for the example to work, we will describe why later. You can now click
On the receiving end of the connection, add a Select CHOP and enter the following expression:
You are receiving over the connection your CHOP channels. Behind the scene, the sender converted the CHOP channels to a NumPy Array, and converted the NumPy Array to a bytearray to be sent over the dataChannel. The receiver does the conversion back from bytearray, to NumPy Array, to CHOP channels. Because we are sending data that can be totally arbitrary and working with bytearrays over the network, this example is not activated by default and values are hardcoded (to support 5 channels of 1 sample) so that the receiver can convert everything back to the CHOP pipeline.
If you wish to adapt this example, you can edit the Palette:webRTC_Ext extension and add a new method named
onDataReceived followed by your channel name. i.e. If we want to add a dataChannel named
Animal, the method name would be
onDataReceivedAnimal. You can then take inspiration on the
onDataReceivedGenericCHOP method and change the shape of the NumPy Array using the NumPy method
Signaling message types specific to WebRTC
Those signaling messages are following the standards as set and detailed in the Palette:signalingServer Signaling API section. They also have matching JSON Schema files.
signalingType - Offer
A signaling message sent from a client with a WebRTC Offer to another client, and transiting through the signaling server.
The signaling message type property should be set to
Offer, and an SDP should be set in the sdp property of the content dictionnary.
"sdp": "The Session Description protocol (SDP) of the local peer following a WebRTC offer"
signalingType - Answer
A signaling message sent from a client with a WebRTC Answer to another client, and transiting through the signaling server.
The signaling message type property should be set to
Answer, and an SDP should be set in the sdp property of the content dictionnary.
"sdp": "The Session Description protocol (SDP) of the local peer following a WebRTC answer"
signalingType - Ice
A signaling message sent from a client with a WebRTC Ice to another client, and transiting through the signaling server.
The signaling message type property should be set to
Ice, and a dictionnary describing a candidate should be set in the sdpCandidate, sdpMLineIndex, sdpMid properties of the content dictionnary.
"sdpCandidate": "The Session Description protocol (SDP) Candidate of the local peer following a WebRTC RTCIceCandidate event"
"sdpMLineIndex": "The index of the media description of the SDP candidate following a WebRTC RTCIceCandidate event"
"sdpMid": "The media stream identification of the SDP candidate following a WebRTC RTCIceCandidate event"
Parameters - WebRTC Page
Signalingclient - The signaling client with which this WebRTC COMP will subscribe to send or receive signaling messages.
Default Tracks Behaviour
Defaulttracksbehaviour - ⊞ - When starting an RTCPeerConnection, initiate the negotiation with default tracks. Various default combinations are available.
STUN Server URL
Stun - URL of the STUN server.
TURN Server URL
Turn0 - URL of the TURN server. See WebRTC#ICE for more details regarding TURN.
Username - Username for access to the specified TURN servers.
Password - Password for access to the specified TURN servers.
Reset - Reset the WebRTC COMP states and kill all ongoing peer connections.
Signaling API Version
Signalingapiversion - The Signaling API version. When a signaling client is being used, the API versions of both the server and client should match.
Parameters - Logger Page
Enablelogging - Enable advanced logging features. Useful to debug and track the WebRTC COMP activity.
Log to File
Logtofile - When enabled, log messages will be written to a time rotated log file.
Logfolder - Path to the folder in which the log files should be written.
Filerotation - Number of backup files to be kept when rotating log.
Log to Textport
Logtotextport - When enabled, log messages will be displayed in the textport.
Log to Status Bar
Logtostatusbar - When enabled, log messages will be displayed in the status bar.
Loglevel - ⊞ - Filter level of log messages. The selected level is included as well as all levels above. I.e. if Warning is selected, Warning and Error messages will be included while Info messages will be excluded.
Parameters - About Page
Help - Open Help page in external browser.
Version - The component version number.