Ever since the VST CHOP came out to the great joy of many audio-visual creators I’ve seen a bunch of tutorials get made on “how to build a sequencer in TouchDesigner” and while offering some great tips and insights they all (including the official VST host palette tool and even my own tutorials) make this one critical mistake. Not updating the sample-rate.
Fortunately, it’s a very easy fix but it does lead us to one of TouchDesigner’s weirdest and most disappointing (for me at least!) bottlenecks: the MIDI in DAT and MIDI in CHOP.
Let's get started!
There are a few approaches to sequencing in TouchDesigner but I would say the most common and my favourite by far relies on the lookup CHOP being driven by a timing ramp of some sort. This allows for a clean and easy workflow with a lot of flexibility.
So let’s say we’ve done just that and create our little sequencer which sends pulses at a regular interval:
We’ve made sure that interpolate was off on the lookup CHOP and in this case, we’ve renamed the pulse channel to correspond to a MIDI note number.
We then plug it into our sound-generating method of choice and listen… What gives?
The sound shuffles around. We increase the speed of our timing ramp to see what’s up and some notes are dropped. It sounds awful and will never work for anything with even remotely strict timing!
Some people at this point might throw their hands up, tell themselves: “TouchDesigner really isn’t usable for musical applications”, end their efforts there and warn their peers not to waste their time.
And they would be wrong! Well… mostly.
The first thing to check for is… are you pulsing parameters like say, “cuepulse” on an audioplay CHOP? (cringe). Remember, the UI and parameter components only update at your network’s FPS so you’ll definitely be hitting a bottleneck there. Same goes for pulsing a trigger CHOP. Here’s a fun network you can use instead if you want to trigger (and retrigger) some one-shots:
Anyways, at this point, all that was left to do was a simple modification to the sample-rate of the timing ramp. In this case, the LFO CHOP.
Just like that: “BAM!” you get super tight sequences and are filled with a hopeful optimism towards your future audio-visual endeavors.
What Happened?
When we first plugged in our LFO to the lookup CHOP, it was set to a resolution of 60fps. This means 1 sample per frame.
This means that we can only ever hope to get an “off to on” message at the beginning of each new frame. The result of which is that timing information in-between is either played a frame early, late, or is lost completely. Using a trail chop set to 8 samples allows us to see the content of all 8 samples of our lookup CHOP.
When we increased our LFO’s sample-rate to 480, we now get 8 samples per channel, per frame! This means that our “off to on” messages can fall anywhere within those 8 samples per frame and increases the accuracy of our timing information. That’s great!
Here we can see that “off to on” signal is happening somewhere within that frame a few samples behind the start of the frame.
What have we learned?
What we’ve learned so far is that multi-sample channels are critical to accurate timing information within our audio-sequencing network.
The MIDI in CHOP Bottleneck:
This brings us to the MIDI in CHOP bottleneck. In short, while the MIDI in CHOP allows you to increase the sample-rate, and thus, the amount of samples per frame, it will treat each sample of the channel identically; If it has received a note-on event within that frame, it will set all samples in that channel for that frame and lose the timing information that those extra samples would offer.
A side note regarding the “preserve pulses” parameter:
As of build 2022.24200 the option to “preserve pulses” was added to the MIDI in CHOP. The wiki reads that:
When on, quick value transitions (pulses) are spaced out over consecutive output samples. Use this option when pulse frequencies approach or exceed the timeline rate, otherwise they risk overlapping each other and being lost.
This to me suggests that the intended use is to record multiple “pulses” per frame such as for time-clock information. It almost seems to want to address the timing problem I am bringing up but not exactly.
I suspect it has to do with time-clock information but I'll be honest in saying that I don’t quite “get it” in the larger context.
Here’s what I mean:
By default, our MIDI in CHOP’s sample rate is set to “me.time.rate” (meaning 60fps in most cases). We’ve established that we want more samples to increase the timing resolution of our messages so let’s set it to 480 as we did our timing ramp:
It reads that while I did manage to increase the amount of samples per second… technically, the amount of samples per frame is still only one. That’s weird. No timing resolution gains.
Now let’s enable the “preserve pulses” parameter:
Our channel now has 8 samples! That’s great! Let’s try it out.
So now let's say I plug in my hardware sequencer, DAW or in this case loop back the efficient little sequencer we’ve built but all we get is the same sloppy timing as before. Ok, so let’s have a look at our samples to see what’s going on:
So we’ve got a sample set to one, a sample set to 0 and then the next sample set to the 0.85. Now I happen to know that I’ve set my sequencer to values of 0.85 only… so where is that value of 1 coming from? Furthermore, on my screen (it can’t really be shown via a picture so you’ll have to take my word for it) I notice that this arrangement of samples repeats almost each frame. If I increase the window length of my trail CHOP, there appears a pulse at the first sample of each multiple of 8 followed by my value of 0.85. Very strange!
What I’ve gathered is that we are still dealing with the issue of all samples being treated identically BUT that "pulse preserve" sneaks in a little 1 to 0 sample at the beginning of our note on event. A “pulse” if you will. I believe that you might be able to get multiple instances of these pulses if you had such a signal to send but this is not something I've tested out.
My take-away is this: The preserve pulse parameter is a bit strange in how it treats incoming MIDI values, has to be set to "on" to output the appropriate amount of samples and makes it even more difficult to figure out what’s up with the MIDI in CHOP.
So what is up with the MIDI in CHOP?
After some back and forth between with the incredibly helpful people of the Derivative team I learned that, to the best of my understanding, MIDI information is not time-stamped, meaning that it only looks at the state of the MIDI messages it receives once per frame and updates the channel accordingly for that frame. The time at which this MIDI information has changed is not recorded and cannot be used to specify at which sample the new value should be updated.
What about the MIDI in DAT?
Whereas the MIDI in CHOP cannot process different values on the same channel per frame, to its advantage, The MIDI in to record ALL events being sent to it which it writes to a table. That could mean multiple note on/offs for the same channel within a frame (I believe the pulse preserve parameter tries to make up for this).
Unfortunately for us once again, this is only evaluated once per frame and the relevant timing information is lost. If we try, for example, to print out “absTime.seconds” per new message received we quickly notice that these time indices are locked in to the start time of the current frame.
What does this mean for VSTis?
As seen in the VST host palette example, note information must be sent to VSTs via DAT executes. Along with the MIDI in DAT, these also execute only once per frame.
While they are no doubt an amazing new feature for TouchDesigner their use for now is still mostly constrained to audio-processing duties lest we make do with wonky timing.
Maybe in the future it might be possible to pass the MIDI information to VSTs directly via CHOPs.
What are my solutions then?
The immediate solution is to crank up the frame rate of our entire network and feel sick to our stomachs for having done such a thing. Ok ok, there are rare times in which this is appropriate (such as when deriving channel information from TOPs) but we've established with our little sequencer that it needn't be necessary.
Alright so it doesn't actually make me feel sick, but it does upset me a little bit considering how close we are to having an amazingly solid and intuitive audio-visual workflow!
Our next solution is to have a dedicated TouchDesigner process with MIDI in and Touch Out rolling at our high FPS :
If I must, I would at least want this process to live comfortably inside an Engine COMP but it doesn’t seem possible to address MIDI devices inside of an Engine COMP for now.
Ok so we’ve got to have this other TouchDesigner process open. It’s not great but I can probably live with it… except that the preserve pulse parameter, which modifies my input values also has to be set to on now for us to gain those 8 samples!
If you're still following this is the point at which I got a bit of a headache.
In Conclusion:
The inclusion of VSTs in TouchDesigner will have people foaming at the mouth looking at this software to build their end-all be-all, interactive, immersive audio-visual dream-machine but given the state of our MIDI input/output situation, it is likely to end in some degree of disappointment and maybe even resignation.
A keyboard player, for example, who needs reliable MIDI timing to accurately capture their performance will still need to load up some other form of software to play their VSTi and do the extra leg-work of finding a pipeline for passing the relevant control information to and from TouchDesigner for their audio-visual masterpiece.
I know this has never been an outright show-stopper, but we are so close to having an amazingly robust audio-visual workflow. It would be shame to not get to the bottom of this MIDI timing thing.
I hope it goes without saying, but this blog post is based on my personal observations to the best of my abilities so if there's anything I've missed please leave a comment and let me know.
TLDR:
Reliable timing in audio sequencing can be easily achieved in TouchDesigner using multi-sample channels but the output or input of sequenced MIDI information is bottlenecked by its MIDI in CHOP’s per-frame channel evaluation.
The “preserve pulse” parameter is a bit strange and complicates things even further.