project file format more suitable for source control

It would be really nice to have TouchDesigner projects and components stored in a format that’s more suitable for source control (like Git or SVN). I have been keeping my work in github repositories ([url]https://github.com/t3kt[/url]), which helps me keep track of what I’m working on and when I changed things.
However, since the whole project is stored in a single binary file, it’s really difficult to see what changed in each revision (without downloading a separate copy of an earlier version and running it in TD). I have looked at the toeexpand/toecollapse tools, but their output is split among a really large number of individual files (many thousands of files for my most recent project). I have split some of my shared components off into individual tox files, but these are just as difficult to work with.

For internal development, I assume that Derivative is using some kind of source control. Since much of the TouchDesigner application is built in TouchDesigner and stored in .tox files, do TD developers just work directly on these binary files?

We develop our UI components the same, using .tox files committed to git. We know this is an issue but it’s still unclear how to cleanly solve it. If it’s 1 file per node, that is likely still too much, no?

It could still just be 1 file per component (like tox files). If they were stored in JSON or XML or some text-based format (except possibly for the parts that need chunks of binary data, like frozen OPs), it would be easier to track changes on them.
The workflow around working with tox files is a bit awkward though. You need to remember to specifically save each individual one any time you make a change, since saving the whole .toe doesn’t seem to save the tox components that are included within it.

If it’s 1 file per node, that is likely still too much, no?

Not necessarily if the changes per node are only those manually made by user - ie the git changesets could be kept small. That wouldn’t work if node internals weren’t constantly changing as a result of some kind of propagation by the editor. I would dearly love to be able to track node changes in git.

One of my partners had a great idea of a check-in/out system where you could take a component, check it out, there by locking it in the main project, but it would still be useable, just nothing would get overwritten inside that component. That way people could continue to work in the main project but any changes they make to the component wouldn’t be saved. Then you would check the component back in, merging all your changes in, and overwriting any changes that may have been made.

That would be fantastic. I haven’t done any TouchDesigner work in a team environment, but I can imagine it gets complicated try to coordinate changes.

I’m bringing this back from the dead. I think in the big picture of TD’s growth in the future is that it needs to become more and more friendly to use in team environments, otherwise it will be overtaken by other tools that are more “enterprise-friendly” or have git integration build into them.

Current trends I’ve seen in adoption of TD works are that a company is excited about TD, wants to build one thing with you to test the relationship, then they’re basically ready to scale up operations and start doing worldwide rollouts of TD projects. This is contrary to what I saw earlier on in my career where everything was essentially a one-off.

So this kind of loops TD full-circle to some more standard roadblocks that software developers have faced and dealt with, a big one being how do a bunch of people work together on one project at the same time. Git being probably the most well known tool. The current TD workflows on teams using git, which most pros can attest to I think, is a mix of walking on eggshells and tyranny (depending on if you’re the team lead or not) combined with a Rube-Goldberg machine someone on your team built that interfaces your TD work with git in variable number of ways.

So whether projects are JSON dumps or some other form of ASCII data, I don’t have a dog in the race. If it’s easier, maybe there could be two TD project files per “project”. One that is the binary data that would be difficult to convert to an ASCII format (which I believe are things like locked operators and vfs and such?) and then a JSON or ASCII representation of all the nodes and parameters. They could work together to load the “full project”, but it could be one method to move forward with this idea without completely inventing the wheel from scratch.

2 Likes

ROFL this is so true

1 Like