Close

Privacy policy

Follow us

https://www.instagram.com/touchdesigner/
https://www.facebook.com/TouchDesigner
https://twitter.com/1null1
https://www.youtube.com/TouchDesignerOfficial
  1. Home
  2. SORA2 INSIDE TOUCHDESIGNER
    Asset

    SORA2 INSIDE TOUCHDESIGNER

    Asset Downloads

    AIMods_Sora2 + PROTOTYPING_UI - V1

     

    Sora-TouchDesigner.png

    After The intégration of Nano Banana here is a new AIMod  : AIMods_Sora2, a TouchDesigner module that brings SORA2 API call into your TD workflow. It supports both TXT2VIDEO and IMG2VIDEO, with a lightweight prototyping UI for rapid experimentation. Clear docs and examples are included, so you can dive in quickly and intuitively.

     

    • AIMOD_SORA2 that integrates the inference API for Sora2 (TXT2VIDEO and IMG2VIDEO).

    • PROTOTYPING_UI: A small UI to test prompts and workflows

    • This Documentation


    FIND THE UPDATE ON MY PATREON : patreon.com/infratonal

    Sora2 QUICK Overview

     

    Sora2 is everywhere right now, so here’s a quick take. Sora2 is OpenAI’s latest video generation and editing model. It supports TXT2VIDEO and IMG2VIDEO and video editing workflows, can extend or remix existing footage. One of its strengths lies in its realistic rendering, semantic and temporal consistency, keeping objects and motion coherent across shots. Since it’s a video model, API calls are naturally heavier than image models, but it's still quite fast. Available resolution are : 1280x720 and 720x1280 and the video lenght could be 4s, 8s or 12s.

    Inside TouchDesigner, Sora2 AIMod can be orchestrated into complex pipelines, combined with other AIMods, or embedded into custom interactive systems. not Easy to use it in Installation and performance, it will maybe be more useful as a preproduction tool. To be honest the model is really impressive on TXT2VID modalities, but to me not as good as something like Kling for IMG2TXT. with a  lot of features missing and warning ...  IT'S A REALLY EXPENSIVE MODEL. And not super reliable yet.

     

    Installing AIMods_SORA2 in TouchDesigner

     

    Create an OpenAI Account

     

    Generate an API Key

     

    Reference the Official Docs and Guides

     

    Once these steps are done, AIMods_SORA2 in TouchDesigner can use your API key to send TXT2VIDEO or video-edit requests to the OpenAI API and fetch the resulting video responses into your TouchDesigner workflow.

     

    Install Your Python Environment & Make it accessible from TouchDesigner

     

    WARNING : THIS PART IS THE SAME YOU MAYBE DONE FOR NANO BANANA AIMODS, SO YOU CAN SKIP IF YOUR ENVIRONEMENT IS ALREDY INSALLED

    To run AI inference from TouchDesigner, you need a Python environment accessible from TouchDesigner. It's highly recommended to use a virtual environment (venv, Conda…) to keep things clean and organized. Each environment is isolated, with its own Python version, libraries, and scripts. This makes it easy to set up different environments for different projects, prevent library conflicts, and make possible to share work setups.

     

    Let's present two methods :

    Method 1 : Install and manage External environment with Anaconda

    It a centralized environments that you install outside TouchDesigner accessible by modifying TouchDesigner's environment variables. This is the 'old' method, but still relevant if you use your environment for other project than those inside TouchDesigner. Note that  it's often recommended to create the environment with the same Python version that ships with your TouchDesigner build. To check which version that is, open the TextPort and DAT : the Python version appears on the first line.

    You can Follow this tutorial to set it up :  https://derivative.ca/community-post/tutorial/anaconda-miniconda-managing-python-environments-and-3rd-party-libraries

    To use this Method in the project

    • This project is configured to use this method by default

    • Once your Conda environement is Installed

    • You can load the environment at startup via op('/project1/START_TOOLS/Load_env').

    • You can configure Conda directly in the CONDA tab of op('/project1/START_TOOLS') (it's somthing like : C:\\Users\\username\\anaconda3\\envs\\Gemini) and reload the project. it will automatically be loaded at each startup.

    • Install your Library under anaconda CLI

     

    Method 2 : Using the TDPYENVMANAGER component (available in experimental 2025 builds)

    It's simpler if you are beginner and it's officially from TouchDesigner. It install a per-project environments in their subdirectory with venv (you can also use conda). it is only available in the Experimental build.

    Here is the tutorial :  https://derivative.ca/community-post/introducing-touchdesigner-python-environment-manager-tdpyenvmanager/72024

    To use the method in the project

    • do not use the /project1/START_TOOLS/load_env script. To do so : Load the project, disable the call to this componant by commenting the line #op('load_env').run() in /project1/START_TOOLS/execute_on_start

    • Save and reload your project

    • Configure your environment with TDPYENVMANAGER

    • Run your environment

    • Install your Library opening Virtual environement CLI

     

    NOTE : This task around Python environement can be a bit tedious and often the cause of many misunderstandings and bugs. However, if you want to seriously use AI models within TouchDesigner, it's important to have a solid grasp of the following concepts: using the CLI (Command Line Interface), creating virtual environments (venv, Conda, etc.), the main Conda commands (for those who use it), installing packages (pip, Conda Forge, uv, etc.), and best practices for maintaining library compatibility. There are plenty of resources online, and your favorite LLM can be a good tutor.


     

    Install Python Library

     

    IN YOUR ENVIRONEMENT CLI (Command Line Interace) : Once your Python environment is set up, you need to install the OPEN AI  library:

    pip install openai

     

    IN YOUR TOUCHDESIGNER TEXT PORT : After installation, you can verify from the Textport that the library is accessible by entering:

    python >>> import openai

     

    If no error appears, it means the library is properly loaded. For final confirmation, you can also check the version:

    python >>> openai.__version__

    '2.2.0'

     

    Configure API Key in TouchDesigner

    You can enter the API key directly in the API_KEY menu of the component Project1/START_TOOL. Save your project and restart it: the key will then be loaded automatically at each startup.

    NOTE : This is not the most secure way to integrate your API key, for a better intégration you can put it in environment variable.

    Simplified Use: PROTOTYPING_UI

     

    For simplified use, I quickly coded a small UI (Like the one I made for Nano Banana), it's quite simple but it does the job. This UI relies on the central node AIMOD_SORA2 (which is completely independent from this UI).

    screenshot_UI.png

     

    You can open the PROTOTYPING_UI component in View mode to use it. It's fairly straightforward, as you'll see:

     

    On the right column

    • You can upload an image (if there is no images) it means you are using TXT2VIDEO mode

      • NONE : does not include any image in the process

      • FILE: loads an image (when you first clic it open the fileselector. As it's a radio button, clic on another option and clic again to reopen the fileselector)

      • LAST: injects image from the last generated video (The tool does not take a reference video as input, but if the chosen reference source is a video, it will grab a screenshot of the video frame at the moment when ‘Start’ is clicked)

      • CAM: uses the camera input (see the configuration of Videodevicein1 in /Project1 for camera setup)

    • Within each source image windows, you can zoom and pan each image to inject the most relevant part for your needs. Double-clicking on an image resets its zoom and position.

     

    Then below you have :

    • Prompt navigation to recall previous prompts
    • Your current prompt input
    • Image navigation to recall previously generated images (handy for injecting a specific image into input images)
    • The Start button sends the request to the API and starts generation : The progress bar shows the processing progress, but even when it reaches 100% the task may not be fully complete and you might need to wait a little longer. If it's too long it means it crashed on server side ;)
    • A small log window displays returned messages and errors

     

    Using the TouchDesigner Component: AIMods_SORA2

     

    To use only the AIMod . Same than for NanoBanana : similar structure, with some variations depending on the modality. The processus is fully asynchronous, but it still depends of your hardware. except for the first launch, TouchDesigner will freeze for a few seconds for loading libraries : this is normal.

     

    Inputs

    • In1 CHOP: External trigger pulse (0 → 1) to start inference

    • In2 DAT: External table with prompt in [0,0] (takes priority over the parameter prompt)

    • In3 TOP: Image1

     

    Outputs

    • Out1 CHOP: Trigger pulse (0 - 1) indicating the end of inference. Useful for chaining the AIMod with other Mods or processes

    • Out2 DAT: External table with text result in [0,0] (if any)

    • Out3 TOP: Result video

     

    Component PARAM Tab

    • Run: Starts inference (same as In1 CHOP)

    • Stop: Forces the termination of the current process

    • Mode: Selects the inference modality

    • Resolution : 1280x720 or 720x1280

    • Duration : 4, 8 or 12

    • User Prompt Input: User prompt (note: this field is ignored if a table is connected to In2 DAT)

     

    File System

     

    This AIMod comes with a small file system, as temporary images are used. You have to set up this two directory in order to make the project working :

    • /Temp: Temporary images (for AIMod)

    • /Export: Saved Videos and images (for PROTOTYPING_UI)

     

    Usage

     

    • When the process is launched (via the Run button in PARAM or the external trigger), the component turns red, indicating it is running in the background.

    • When the process receives a response from the API, outputs are updated, Out1 CHOP pulses, and the component returns to its normal color.

    • A running process cannot be relaunched until it has been stopped (this is very useful for agent-based orchestration).

    • At launch, the module analyzes connected TOP inputs and automatically selects the modality depending on the connections:

      • If no images are connected : TXT2VID is called.

      • If there is an image connected IMG2VID is called.

    • If the process encounters an error, it should display a message in the PARAM tab. then STOP the background process, and return to normal color.

    • BUT, this doesn't always happen. Sometimes, in case of certain errors, the node may remain red and you'll need to manually stop the process by pressing STOP before launching a new one.

     

    Note

    • The API is not super reliable

    • it's happen multiple time the API return get stuck at the end of the process (I've a little script to check the status of a video ID inside the AIMods) ... In this case you'll have to STOP manually the AIMod

    • Sometime the vertical format 720x1280 generate an error ... "process failed"

    • I leave the ChatGPT 'gpt-image-1' model in the selection, you can use from the AIMod menu, but not very well integrated with the UI

     

    Prompting Guide

     

    License of AIMods_SORA2 + PROTOTYPING_UI

    • This code is provided as is (aka AS IS).

    • It may crash or behave oddly… that's life.

    • Copyright (c) 2025 Infratonal

    • Licensed under the MIT License

     

    Asset Downloads

    AIMods_Sora2_V1.zip (36.89 KB)

    Comments