Introducing the TDPyEnvManager COMP
TDPyEnvManager
COMP is designed to become your homebase when working with sideloaded Python environments and third party libraries. We aim at making it the trusted go-to solution for TouchDesigner users and advanced developers.TDPyEnvManager
COMP is a custom component that helps manage Python environments (vEnv) as well as Conda environments, the primary focus is for sideloading environments and integrating third party libraries more easily in TouchDesigner.!!! Sideloading Python environments and third party packages is at your own risk and can cause instabilities. For more details, head to https://docs.derivative.ca/Python
vEnvs
and conda
environments from the component's parameter dialog. Users can interact directly with pip
in both vEnvs
and conda
CLIs, as well as conda
commands when using conda
.!!! With the complexity of covering multiple systems (ie. Windows, MacOS, and environments from Python and Conda), comes the complexity of testing. Some edge cases might not be covered and we are inviting the community for feedback. If you find that you are missing a feature, or that an implemented feature might need a change or some UX improvements, feel free to send us a message in the forum.
Using TDPyEnvManager with Python vEnv
TDPyEnvManager
. While we could support management of multiple vEnvs, the idea here is to enable for only one vEnv
per project folder with a limited set of features for a common and streamlined approach.Let’s get started.
MyTDEnvTest
.MyTDEnvTest
a .toe
file named MyTDEnvTest.toe
. Then open this new .toe
file. Delete any OPs in /project1
.Active
toggle. This toggle is handy when you have already setup an environment but you want to prevent it from being added to the TouchDesigner search path on startup for debugging purposes or prototyping.You’ll see a dialog with a disclaimer stating that sideloading environments and using third party libraries in TouchDesigner is at your own risk. You can click continue if you are fine with the disclaimer.
Python vEnv
to be streamlined and straightforward, you can simply click on Create vEnv
to get going.vEnv
will be following the same naming schemes on all machines. It will use the project folder name and add the suffix _vEnv after it.MyTDEnvTest_vEnv
will be created in MyTDEnvTest
next to MyTDEnvTest.toe
.TDPyEnvManager
, as well as the Status
parameter in the parameter dialog updating.vEnv
will be done and you should see in the Status
the message Environment linked and ready.
Open CLI
: it will open a new terminal, with the vEn
v you just created being activated.pip install zmq
. It will look for the zmq
package on PyPi
and download and install it in the vEnv
.Alt+T
), type import zmq
followed by zmq
. It will import zmq
and show in the textport the zmq
module as well as where it’s loaded from.zmq
was loaded from wherever you created your MyTDEnvTest
folder: MyTDEnvTest/MyTDEnvTest_vEnv/Lib/site-packages/zmq/__init__.py
! A note about UV - If trendy UV is your preferred tool, you can install it within yourvEnv
usingpip install uv
You can then useuv your_command
in the running CLI.
Sharing my vEnv setup with others
Export requirements.txt
requirements.txt
file next to your currently running .toe
file.!!! Important note: This is not an optimalrequirements.txt
file. You need to go through it manually and curate it to avoid setting too many version restrictions and possibly add a--extra-index-url <https://some.url>
line within therequirements.txt
file.
requirements.txt
with another user.requirements.txt
file next to the .toe
file they have open, and when they click on Create from requirements.txt
it will create a vEnv
and install all the required packages, such as zmq
.! If a matching folder of the same name is already present, the folder doesn’t get cleared. The environment gets created if a file is missing in the target folder, but an existing environment will be unaffected unless there is a change in dependencies caused by the Python builtins or a possiblerequirements.txt
file being present. This is valid for both creating a barebonevEnv
from TouchDesigner or creating avEnv
with arequirements.txt
file being present.
Using TDPyEnvManager with Conda
Python vEnv
approach. We are using conda
and the Miniconda installer. On top of allowing for conda
packages to be installed (as well as pip
packages), we can specify the conda
install folder in which environments will be looked for and created, we can create multiple conda
environments, and list conda
environments that are installed on the machine globally. Similar to the Python vEnv
approach, we can export our environment.yml
files and create environments from environment.yml
files.!!! When using theconda
mode, TouchDesigner is downloading and installing the Miniconda installer in the background. When silently installing the Miniconda installer, users agree to the Anaconda Inc. ToS: https://legal.anaconda.com/policies/en/TouchDesigner adds restrictions so that environments created are setup to fetch packages from conda-forge channels, which are not the default channels covered by the Anaconda Inc. ToS. However, there is a probability for user or system configurations to take priority and the restrictions might not always apply.
Creating and activating a conda
environment
.toe
file.TDPyEnvManager
Active
parameter on, and click Continue
in the popup with the disclaimer.Conda Env
in the Mode
parameter.Conda Install Folder
parameter as is.MySuperCondaEnv
.Create Conda Env
..toe
file. Once it’s downloaded, it will start the installation.! If you are working offline and already downloaded the Miniconda installer, place it next to your.toe
file and click on the toggleKeep Conda Installer
. It will re-use the installer and keep it after install. On Windows, you can find the installer at the following link: https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe. Keep it namedMiniconda3-latest-Windows-x86_64.exe
. On MacOS, you can find the installer at the following link: https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh. Keep it namedMiniconda3-latest-MacOSX-arm64.sh
.
Continue
to proceed.conda
will proceed and create a new environment.Now, you can proceed and click on the Open CLI
pulse parameter to open the conda
prompt with your activated environment.
MySuperCondaEnv
, I see the CLI input being preceded by the (MySuperCondaEnv)
which indicates which environment is activated.conda
commands, such as conda install
as well as pip
commands.tqdm
with conda install tqdm
.import tqdm
followed by tqdm
and see that my package is properly installed and imported from my MySuperCondaEnv
environment.Listing global environments
conda
install, toggling on the Include Global
parameter will list all environments on the machine with a Python version matching TouchDesigner’s. Global environments are in the dropdown menu of the Environment
parameter and are suffixed by a star *
.Sharing my conda env
setup with others
Export environment.yml
and an environment.yml
will be created, relative to your .toe
file. The difference with a Python requirements.txt
file is that it includes the Python version as a requirement to the environment created, the environment name, as well as packages of various sources and the channels in use.environment.yml
file next to the .toe
file they are currently using and click on the Create from environment.yml
pulse parameter.!!! Note that if a matching environment already exists, it will be overridden by the environment.yml requirements.
Advanced usage
TDPyEnvManager
programmatically using its extension promoted attributes and methods, or you can use the TDPyEnvManagerHelper
directly. The TDPyEnvManagerHelper
can be used as a module and imported from the TDPyEnvManager COMP/TDPyEnvManagerHelper
or as an attribute of the TDPyEnvManager
itself, Helper
.Samples/TDPyEnvManager
folder for samples.Considerations
TDPyEnvManager
in your projects, and when custom components are relying on third party libraries loaded from side loaded environments, it is preferable that you let the initialization of custom components being handled lazily.Init On Start
toggle.TDPyEnvManager
should be the first component initializing on start so that the environment is setup before any additional component attempt to import modules from it.try / except
condition around your import
statements at the top of your extension.try: import tensorrt as trt import torch import torchvision.transforms as transforms except Exception as e: logger.error(f'TDDepthAnythingHF - An error occured trying to import some of the required libraries. Make sure that the environment is setup properly.') logger.error(f'TDDepthAnythingHF - {e}')
When setting up your TDPyEnvManager
and side environment, you can safely ignore issues from custom components. When your environment is setup properly and you installed the required dependencies, restart TouchDesigner and any import issue should be fixed.
Tips
{ "folders": [ { "path": "." } ], "settings": { "python.defaultInterpreterPath": "C:\\Program Files\\Derivative\\TouchDesigner.2023.32465\\bin\\python.exe", "python.analysis.autoSearchPaths": true, "python.autoComplete.extraPaths": [ "C:\\Program Files\\Derivative\\TouchDesigner.2023.32465\\bin\\python\\Lib\\site-packages", ".\\scripts", ".\\TDDepthAnythingHF_vEnv\\Lib\\site-packages", ], "python.analysis.extraPaths": [ "C:\\Program Files\\Derivative\\TouchDesigner.2023.32465\\bin\\python\\Lib\\site-packages", ".\\scripts", ".\\TDDepthAnythingHF_vEnv\\Lib\\site-packages", ], } }
In the example above, the interpreter is pointing to TouchDesigner’s own Python binary and we added TouchDesigner site-packages
as well as a Python vEnv
plus a relative scripts
folder.
Conclusion
TDPyEnvManager
and the Thread Manager
to work with the Hugging Face transformers library and Depth Anything V2.