Anaconda - Managing python environments and 3rd-party libraries in TouchDesigner
It has been quite a few times that I see on the Derivative forum or on social networks, cases where users are struggling with third party Python libraries / packages integration in TouchDesigner. While you should not consider the following example the ultimate solution, it saved me quite a few times and Anaconda is a nice tool to use even outside of the TouchDesigner context.
Important: Mac users, see comments and visit https://docs.derivative.ca/Category:Python#MacOS
Important note: When adding your own version for a package that is already shipped with TouchDesigner, you might encounter unexpected behaviors.Many of our internal tools and palette components rely on NumPy and/or OpenCV. Loading different versions of Numpy and/or OpenCV is at your own risk. Some other issue could be with the following: considering a Package A with a dependency B, if updating your sys.path cause a different version of dependency B to load first, it could cause issues with Package A.
What is Conda / Anaconda?
Conda is a cross-platform, language-agnostic binary package manager. It is the package manager used by Anaconda installations, but it may be used for other systems as well. Conda makes environments first-class citizens, making it easy to create independent environments even for C libraries. Conda is written entirely in Python, and is BSD licensed open source.
As per https://github.com/conda/conda
What is the difference between Anaconda and Miniconda?
Miniconda is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others.
As per https://docs.conda.io/en/latest/miniconda.html
For the sake of this tutorial, at the time of writing (06 16 2021), the version used are:
TouchDesigner 2021.13610, the latest stable release
Anaconda install is Anaconda3-2021.05-Windows-x86_64 Python 3.8 Win 10 64-bit
Windows 10 64-bit
The install is fairly straightforward. For the sake of that example, we will do a vanilla install of the latest version of Anaconda (Windows 64-bit) in a fairly vanilla environment, meaning: no local Python installation, no changes, no previous installation of Anaconda or other things. It’s a pretty clean environment.
First, head to the Miniconda documentation page or the Anaconda (full installer) page.
What you’ll want is to download the installer for Windows / Mac, preferably in 64-bit and (important) w/ Python 3.x. The latest builds of either Anaconda or Miniconda 3 should all be coming with Python 3.8 by default.
Once the installer is downloaded, start the installation, and go through the install steps. You can keep all the default recommended choices.
You are done with the installation, let’s check that everything is installed and running correctly.
Let’s go over this screenshot and a basic conda command.
In the terminal, you see
(base) which is here to remind you which environment of conda is currently the active environment. Followed by your user path.
The first command we’ll learn is
conda env list which is pretty self-explanatory, it will list your environments.
You should get a result similar to the screenshot above, where the star
* before the path is the currently active environment.
If you get a similar result, we are good to go.
Create an environment
First, launch TouchDesigner.
You want to know which version of Python TouchDesigner is currently using so that you can match this version in your conda environment.
To do so, use Alt+T in TouchDesigner, which will open your textport.
You’ll see the following window.
You can read on the second line that the Python version currently used by TouchDesigner is Python 3.7.2.
We will match this version.
Now, go back to the conda command prompt and type:
conda create -n td-demo python=3.7.2
conda - the shortcut/context for Anaconda
create - self-explanatory, to create an environment
-n - for the name of the environment followed by your new environment name, with no space, here “td-demo”
python= - to force a python install in the version matching TouchDesigner, 3.7.2 in our case
I like to name my TouchDesigner environments with “td-” since I work a lot with Anaconda, sometimes outside of the TouchDesigner context. It helps me avoid using an environment for, let’s say, Tensorflow, and using that same environment in the td- context w/ a bunch of other packages that might cause issues with a side project or just be confusing.
This is quite important because environments/environment management is one of the great features of Anaconda. And you want to avoid messing with your TouchDesigner local python installation or using a conda environment with a Python version that doesn’t match TouchDesigner.
Once you type the command, press enter, it might take a little while.
You’ll see the following lines (or similar) printing in the command prompt.
It is the default wheels and binaries for the conda environment to run in Python 3.7.2. You can press
y to proceed.
Conda will download and install all the packages in your new environment.
Once the downloads and installation are done, you’ll see the following (or similar).
Activate (move to) an environment
As it is stated in the screenshot above, you can now use the command
conda activate td-demo to move to this newly created environment.
activate is the action and
td-demo the environment we want to move to. You should now see at the front of the line
(td-demo) which means you are in your new environment.
Install additional packages
For the sake of this tutorial, we are going to install 2 main packages.
The first one is sckikit-learn. A recent question in the forum made me go through these exact same steps to debug a user issue, and it was extremely easy to get scikit up and running in TouchDesigner with conda. The second one is open3d, which was introduced by Darien Brito recently, in his Kinect Azure Point Cloud merger.
Let’s go through the steps on how to install those packages.
Most of the time, typing
conda install packagename will work. But a good source to find packages, package names and versions is to go to the central repository on https://anaconda.org/
Note: pip does come with Anaconda, if a package is missing from the Anaconda package repository, you can still use pip with the more traditional
pip install packagename. It will still install the package, using pip this time, but as part of your Anaconda environment.
Let’s look for Scikit, and we can easily find out that the command we are looking for is
conda install scikit-learn
You can now type the following in your conda command prompt, and conda will look for all the required wheels and binaries for scikit-learn to run properly, in your current environment and on Python 3.7.2.
You should see the following
You can press
y to install the extra packages.
Now, let’s do the same with open3d. On anaconda.org, we can find that the command to use is
conda install -c open3d-admin open3d
Let’s type that and press enter in our conda command prompt.
You should see quite the list of additional packages and dependencies. If you don’t need open3d, you can press
n, else, process with the install and press
Additionally, you can use the command
conda list within your environment to list all the packages installed.
Link environment to TouchDesigner
Welcome to the TouchDesigner side, it might sound tricky, but it’s actually fairly straightforward here.
Open your TouchDesigner project.
In this project, you will create an Execute DAT and toggle on the “Start” toggle parameter.
What we want to do next, on startup, ensure that the site-packages of our conda environment are added to the path of TD and packages binaries / libs / dlls.
First, let’s type in the Anaconda command prompt the command we used earlier
conda env list
Remember? It’s listing the environments and paths of each environment.
Look for your newly created environment, or the active (the one with a star *) environment you want to add to TouchDesigner.
Write down the path, if your install is using default settings, it should be something like
In our case, if you are following this tutorial, the environment name should be
Back to TouchDesigner, at the top of the Execute DAT, replace the
onStart() method with the following code
Note: Double check all the paths used in the code snippet, it could be that your anaconda 'envs' folder is not in your User folder depending on your conda install settings and conda version, as well as your OS.
import sys import os import platform def onStart(): user = 'YOUR_USERNAME' # Update accordingly condaEnv = 'YOUR_ENVIRONMENT_NAME' # Update accordingly if platform.system() == 'Windows': if sys.version_info.major >= 3 and sys.version_info.minor >= 8: """ Double check all the following paths, it could be that your anaconda 'envs' folder is not in your User folder depending on your conda install settings and conda version. """ os.add_dll_directory('C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/DLLs') os.add_dll_directory('C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/Library/bin') else: """ Double check all the following paths, it could be that your anaconda 'envs' folder is not in your User folder depending on your conda install settings and conda version. """ # Not the most elegant solution, but we need to control load order os.environ['PATH'] = 'C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/DLLs' + os.pathsep + os.environ['PATH'] os.environ['PATH'] = 'C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/Library/bin' + os.pathsep + os.environ['PATH'] sys.path = ['C:/Users/'+user+'/miniconda3/envs/'+condaEnv+'/Lib/site-packages'] + sys.path else: """ MacOS users should include path to .dlybs / MacOS binaries, site-packages """ os.environ['PATH'] = '/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/lib' + os.pathsep + os.environ['PATH'] os.environ['PATH'] = '/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/bin' + os.pathsep + os.environ['PATH'] # The following path might need editing (python3.9) based on the python version used with conda sys.path = ['/Users/'+user+'/opt/miniconda3/envs/'+condaEnv+'/lib/python3.9/site-packages'] + sys.path return
YOUR_WINDOWS_USERNAME should be the name of your user folder, found in
YOUR_ENVIRONMENT_NAME should be
td-demoif you are following this tutorial.
You can now either save your project and restart or simply pulse the pulse button of the Start parameter of the Execute DAT.
os.add_dll_directory()which is quite handy in our use case, but not yet available in Python 3.7.2 at the time of writing of this article.
A note for MacOS users:
To know where is your Anaconda environment, as per Anaconda’s official doc:
- Open a terminal window.
- If you want the location of a Python interpreter for a conda environment other than the root conda environment, run
conda activate environment-name.
In the same terminal, you can type
python, and import a package such as
import numpy if you installed numpy in that environment. Typing
numpy, it will print out where was numpy imported from and show the site-packages folder location, within your Anaconda environment folder.
IMPORTANT: You might encounter issues if you are on an M1 Mac, and run different architectures. If you are running the native ARM build of TouchDesigner, your Anaconda install, Anaconda environment and libraries should all be the native ARM versions as well. More details are available with documentation for Homebrew here: Category:Python - Derivative
Use your packages (and environment) in TouchDesigner
You should now be able to import the packages you installed using conda. Let’s make sure of it. In TouchDesigner, press Alt+t to open the textport.
Starting with scikit-learn, type
import sklearn and press enter. If the command goes through without error, it means sklearn was imported with success.
You can make sure by typing
sklearn.__file__ which should point to the
__init__.py of sklearn or
sklearn.__version__ which should print the same version that the Anaconda command prompt was listing.
If you installed open3d as well, you can proceed with the same command
import open3d. If the command goes through without error.
You can now use those packages in TouchDesigner, congratulations!
If an issue occurs, it can be due to a large number of factors. I would recommend first to go back to your environment in the conda command prompt and type
python -c “import sklearn”, if the command is working without issues, then an issue occurred while linking your conda environment to TouchDesigner.
You now have a basic introduction to Anaconda and a simple integration in TouchDesigner. It should help unlock access to a few interesting projects with all the great libraries that come out from the Python community. I hope that this tutorial helped and that you were able to go through it without issues. If you have any questions, comments, points to add, please leave a comment and I will try to answer accurately and update this tutorial if necessary!