Anaconda - Managing python environments and 3rd-party libraries in TouchDesigner
Note: This article was updated on November 7th 2024 to reflect the latest versions. We also simplified the process by installing and mentioning only Miniconda from the installation section and forward.
It has been quite a few times that I see on the Derivative forum or on social networks, cases where users are struggling with third party Python libraries / packages integration in TouchDesigner. While you should not consider the following example the ultimate solution, it saved me quite a few times and conda is a nice tool to use even outside of the TouchDesigner context.
Important: Mac users, see comments and visit https://docs.derivative.ca/Category:Python#MacOS
Important note: When adding your own version for a package that is already shipped with TouchDesigner, you might encounter unexpected behaviors.Many of our internal tools and palette components rely on NumPy and/or OpenCV. Loading different versions of Numpy and/or OpenCV is at your own risk. Some other issue could be with the following: considering a Package A with a dependency B, if updating your sys.path cause a different version of dependency B to load first, it could cause issues with Package A.
What is Conda / Anaconda?
Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for other systems as well. Conda makes environments first-class citizens, making it easy to create independent environments even for C libraries. Conda is written entirely in Python, and is BSD licensed open source.
As per https://github.com/conda/conda
What is the difference between Anaconda and Miniconda?
Miniconda is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.
As per https://docs.conda.io/en/latest/miniconda.html
Setup
For the sake of this tutorial, at the time of writing (Nov 7th 2024), the version used are:
-
TouchDesigner 2023.12000 the latest stable release
-
Miniconda install is latest (24.9.2) Miniconda3-latest-Windows-x86_64 Python 3.12.7 Win 11 64-bit
-
Windows 11 64-bit
Install MINICONDA
The install is fairly straightforward. Follow the steps as described here: https://docs.anaconda.com/miniconda/miniconda-install/
For the sake of that example, we will do a vanilla install of the latest version of Miniconda (Windows 64-bit) in a fairly vanilla environment, meaning: no local Python installation, no changes, no previous installation of conda or other things. It’s a pretty clean environment.
First, head to the Miniconda documentation page.
What you’ll want is to download the installer for Windows / Mac, preferably in 64-bit and (important) w/ Python 3.x. The latest builds of Miniconda 3 should be coming with Python 3.x by default.
Once the installer is downloaded, start the installation, and go through the install steps. You can keep all the default recommended choices.
You are done with the installation, let’s check that everything is installed and running correctly.
Confirm installation
Let’s go over this screenshot and a basic conda command.
In the terminal, you see (base) which is here to remind you which environment of conda is currently the active environment. Followed by your user path.
The first command we’ll learn is conda env list which is pretty self-explanatory, it will list your environments.
You should get a result similar to the screenshot above, where the star * before the path is the currently active environment.
If you get a similar result, we are good to go.
Create an environment
First, launch TouchDesigner.
You want to know which version of Python TouchDesigner is currently using so that you can match this version in your conda environment.
To do so, use Alt+T in TouchDesigner, which will open your textport.
You’ll see the following window.
You can read on the second line that the Python version currently used by TouchDesigner is Python 3.11.1.
We will match this version as close as possible.
Now, go back to the conda command prompt and type:
conda create -n td-demo python=3.11
Where:
conda - the alias for Conda
create - self-explanatory, to create an environment
-n - for the name of the environment followed by your new environment name, with no space, here “td-demo”
python= - to force a python install in the version matching TouchDesigner, 3.11.x in our case
I like to name my TouchDesigner environments with “td-” since I work a lot with Conda, sometimes outside of the TouchDesigner context. It helps me avoid using an environment for, let’s say, Tensorflow, and using that same environment in the td- context w/ a bunch of other packages that might cause issues with a side project or just be confusing.
This is quite important because environments/environment management is one of the great features of Conda. And you want to avoid messing with your TouchDesigner local python installation or using a conda environment with a Python version that doesn’t match TouchDesigner.
Once you type the command, press enter, it might take a little while.
You’ll see the following lines (or similar) printing in the command prompt.
It is the default wheels and binaries for the conda environment to run in Python 3.11.x. You can press y to proceed.
Conda will download and install all the packages in your new environment.
Once the downloads and installation are done, you’ll see the following (or similar).
Activate (move to) an environment
As it is stated in the screenshot above, you can now use the command conda activate td-demo to move to this newly created environment.
Where activate is the action and td-demo the environment we want to move to. You should now see at the front of the line (td-demo) which means you are in your new environment.
Congrats!
Install additional packages
For the sake of this tutorial, we are going to install 2 main packages.
The first one is sckikit-learn. A recent question in the forum made me go through these exact same steps to debug a user issue, and it was extremely easy to get scikit up and running in TouchDesigner with conda. The second one is open3d, which was introduced by Darien Brito recently, in his Kinect Azure Point Cloud merger.
Let’s go through the steps on how to install those packages.
Most of the time, typing conda install packagename will work. But a good source to find packages, package names and versions is to go to the central repository on https://anaconda.org/
Note: pip does come with Conda, if a package is missing from the Conda package repository, you can still use pip with the more traditional pip install packagename. It will still install the package, using pip this time, but as part of your Conda environment.
Let’s look for Scikit, and we can easily find out that the command we are looking for is conda install scikit-learn
You can now type the following in your conda command prompt, and conda will look for all the required wheels and binaries for scikit-learn to run properly, in your current environment and on Python 3.11.x.
You should see the following
You can press y to install the extra packages.
Now, let’s do the same with open3d. On anaconda.org, we can find that the command to use is conda install -c open3d-admin open3d
Let’s type that and press enter in our conda command prompt.
You should see quite the list of additional packages and dependencies. If you don’t need open3d, you can press n, else, process with the install and press y.
Additionally, you can use the command conda list within your environment to list all the packages installed.
Link environment to TouchDesigner
Welcome to the TouchDesigner side, it might sound tricky, but it’s actually fairly straightforward here.
Open your TouchDesigner project.
In this project, you will create an Execute DAT and toggle on the “Start” toggle parameter.
What we want to do next, on startup, ensure that the site-packages of our conda environment are added to the path of TD and packages binaries / libs / dlls.
First, let’s type in the miniconda command prompt the command we used earlier
conda env list
Remember? It’s listing the environments and paths of each environment.
Look for your newly created environment, or the active (the one with a star *) environment you want to add to TouchDesigner.
Write down the path, if your install is using default settings, it should be something like C:/Users/YOUR_USERNAME/miniconda3/envs/YOUR_ENVIRONMENT_NAME
In our case, if you are following this tutorial, the environment name should be td-demo
Back to TouchDesigner, at the top of the Execute DAT, replace the onStart() method with the following code
Note: Double check all the paths used in the code snippet, it could be that your conda 'envs' folder is not in your User folder depending on your conda install settings and conda version, as well as your OS.
import sys import os import platform def onStart(): user = 'YOUR_USERNAME' # Update accordingly condaEnv = 'YOUR_ENVIRONMENT_NAME' # Update accordingly if platform.system() == 'Windows': """ Double check all the following paths, it could be that your conda 'envs' folder is not in your User folder depending on your conda install settings and conda version. """ os.add_dll_directory('C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/DLLs') os.add_dll_directory('C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/Library/bin') sys.path = ['C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/Lib/site-packages'] + sys.path else: """ MacOS users should include path to .dlybs / MacOS binaries, site-packages """ os.environ['PATH'] = '/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/lib' + os.pathsep + os.environ['PATH'] os.environ['PATH'] = '/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/bin' + os.pathsep + os.environ['PATH'] # The following path might need editing (python3.9) based on the python version used with conda sys.path = ['/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/lib/python3.9/site-packages'] + sys.path return
Where:
-
YOUR_WINDOWS_USERNAME should be the name of your user folder, found in
C:/Users/ -
YOUR_ENVIRONMENT_NAME should be
td-demoif you are following this tutorial.
You can now either save your project and restart or simply pulse the pulse button of the Start parameter of the Execute DAT.
To know where is your conda environment, as per Conda’s official doc:
- Open a terminal window.
- If you want the location of a Python interpreter for a conda environment other than the root conda environment, run
conda activate environment-name. - Run
which python.
In the same terminal, you can type python, and import a package such as import numpy if you installed numpy in that environment. Typing numpy, it will print out where was numpy imported from and show the site-packages folder location, within your Conda environment folder.
IMPORTANT: You might encounter issues if you are on an M1 Mac, and run different architectures. If you are running the native ARM build of TouchDesigner, your Conda install, Conda environment and libraries should all be the native ARM versions as well. More details are available with documentation for Homebrew here: Category:Python - Derivative
Use your packages (and environment) in TouchDesigner
You should now be able to import the packages you installed using conda. Let’s make sure of it. In TouchDesigner, press Alt+t to open the textport.
Starting with scikit-learn, type import sklearn and press enter. If the command goes through without error, it means sklearn was imported with success.
You can make sure by typing sklearn.__file__ which should point to the __init__.py of sklearn or sklearn.__version__ which should print the same version that the Anaconda command prompt was listing.
If you installed open3d as well, you can proceed with the same command import open3d. If the command goes through without error.
You can now use those packages in TouchDesigner, congratulations!
If an issue occurs, it can be due to a large number of factors. I would recommend first to go back to your environment in the conda command prompt and type python -c “import sklearn”, if the command is working without issues, then an issue occurred while linking your conda environment to TouchDesigner.
Conclusion
You now have a basic introduction to Conda and a simple integration in TouchDesigner. It should help unlock access to a few interesting projects with all the great libraries that come out from the Python community. I hope that this tutorial helped and that you were able to go through it without issues. If you have any questions, comments, points to add, please leave a comment and I will try to answer accurately and update this tutorial if necessary!