We recently posted an article about an opera and a ballet being staged with 3D stereo-vision virtual sets for the first time. Andrew Quinn working with 3DLive Live was responsible for this landmark achievement and used TouchDesigner to develop the 3D technology employed in this way for the first time.
Andrew, one of the first TouchDesigner users, is based in Milan where he teaches a TouchDesigner course as part of a masters program in Digital Environment Design at NABA. The masters program was founded in 2007 by Paolo Atzori with the goal of providing a 360° view in the field of Interaction and Digital Environment Design and to evolve the technical and critical skills needed to create environments that integrate the digital scenarios of the information society in new representations of space characterised by dynamics of pervasiveness and interaction.
Andrew and teaching assistant graduate student Nima Gazestani have for some time now produced impressive live audio-reactive visuals to accompany classical music performances. They have also graduated proficient and inventive TouchDesigner users from the Digital Environment Design program over the past three years.
You can read about how TouchDesigner was used to make opera 3D here. This article focuses on the teaching and learning of TouchDesigner at NABA and the rather facinating work produced by students and faculty. What follows are interviews with Andrew, Nima and graduate students Carolina Travi and Elena Castellini who were very generous in sharing their work and experiences. We hope it's both valuable and inspiring to would-be TouchDesigner students and instructors.
ANDREW QUINN, Professor + Realtime Media Artist
Derivative: Andrew, please tell us a bit about your background-- what you do and the tools you use.
Andrew: My background is in 3D animation. I started with Side Effects PRISMS in the late 1980s and graduated to Houdini when it first came out in ‘96 while working on Alex Proyas’ Dark City. Compositing is also part of my expertise, as it is with most professional 3D artists. I’ve made a number of short films and enjoy very much the editing process.
I have also worked as a professional musician, and composed music using digital tools so I found TouchDesigner to be a familiar environment, taking my 3D expertise and knowledge of music back into performance.
D: Tell us about some of the motivations and concepts driving what you do.
AQ: The thing that motivates me in my work with TouchDesigner is the merging of sound and image, creating imagery that is sound-reactive. I have always been fascinated by the counterpoint between music and image and TouchDesigner provides the toolset to explore a direct relationship between music and imagery generated in real time.
D: You have been teaching TouchDesigner for a few years. What has the process been? How do you teach it and how do students learn?
AQ: I have been teaching a 30 hour TouchDesigner course as part of a masters in Digital Environment Design at NABA in Milan. The course covered basic TouchDesigner training including some 3D. This was followed by 20 hours of workshop in which the students prepared material for a series of concerts of contemporary music at Milanís Teatro dal Verme, projected in full HD.
Students worked in teams or alone to produce sound reactive pieces for each concert. Some students were already experienced in realtime performance graphics using other software packages so they were equipped to think in this environment and were quickly operative. Others needed more support and were given components as a basis. Each performance included a piece of mine, and also for some pieces we used a simple component using 8 spots with projection maps that reflected the lighting setup of the projection area behind the orchestra in the concert hall. This allowed us to fill the entire 1hour programing the limited time available.
I found that the students who were interested learnt quickly and soon became reasonably self sufficient. There was a lot of emphasis on CHOPS and I found it essential to emphasize the need to calibrate the Math CHOP! Also basic working practices for example, exporting from a null CHOP. I emphasized the organization of work, layout, use of self-sufficient components for sharing, labeling in and out components and text DATs for documentation.
D: What has your experience been teaching TouchDesigner? Are there identifyable stumbling blocks?
AQ: Obviously it is difficult to teach 3D to those with no previous knowledge. Surprisingly, many had problems understanding the procedural node-based aspect, the concept of the flow of information.
Something as simple as the trigger CHOP, the concept of ADSR (attack decay sustain release) is totally unfamiliar to those who, for example, have never worked with keyboards. It’s not a bad idea to bring a small keyboard into the classroom and explain aurally the concept.
The other thing that intimidates some students is the use of expressions, even the $F variable... these concepts need to be introduced slowly. The concept of Environment Variable for example is alien to many. Students arrive at all levels of prior knowledge and all need to be accommodated so it was difficult to estimate the pace of teaching.
D: How was TouchDesigner integrated into the course? Was it used with other software tools?
AQ: TouchDesigner was used with MaxMSP, connected via OSC for some of the concert performances. Also Max was used for some of their interactive pieces along with Mocolo, an Italian software for presence and motion detection, written by one of our lecturers, again connected to TouchDesigner via OSC.
D: With its visual architecture and WYSIWID (what you see is what its doing) nature, how does TouchDesigner assist in the scope of teaching and learning?
AQ: I find TouchDesigner very useful for the visualization of sound - helping students to see a waveform and then decide how to utilize it in a visual way. It is also very interesting for them to apply mathematical operations to 3d geometry, the torus SOP -> noise SOP for many is a revelation. Using CHOPs for the same end is more complex to grasp but is the basis for a deeper understanding of the possibilities of geometry manipulation. Also in the 3D context, the view from a light is an enormous help to students to understand and visualize lighting in 3D. I found the particle SOP difficult to teach, establishing initial velocities etc. Again students were given a template to work from.
D: What next? What are your future plans?
AQ: We are planning a 40hour night course in TouchDesigner at NABA for October which will be open to the general public. It will be interesting to see what response we have. Students from this course will be given the opportunity to perform in the contemporary music festival Koine2012. Next year I am doing the digital sets for an new opera by Ivan Fedele to be premiered at a summer music festival in Italy, again real time, sound reactive visuals.
My main emphasis in the use of TouchDesigner is the sound reactive capability. From my experience performing such material in the contemporary music field, I can say with confidence that the perfect synchronization between music and visual is still an undiscovered territory and something I am very excited in exploring with interested musicians. Performed with subtlety, it can enhance and provide an emotion to the music without competing with the composition
CAROLINA TRAVI, 3D + Environmental Designer
I was Andrew Quinn’s student at NABA in the year 2009, and I’ve been using TouchDesigner ever since, although not as much as I would like to.
I’m from Buenos Aires, Argentina. Every work that I do I propose to use it, but only a few have accepted because very little people know about it. But in my personal projects is the most important software I use, for visuals or interactive audiovisual installations. You can see more on my website.
D: Please tell us a bit about your background-- what you do and the tools you use.
CT: I’ve studied at the Universidad Del Salvador in Buenos Aires and received an Honors Degree in Theater Arts in 2008, specialized in 3D design. I also worked in independent plays for theaters as a set designer and costume designer in the city of Buenos Aires. In the 3D area she developed projects for several architect’s studios and producers.
In 2009 I traveled with a scholarship to the city of Milan, Italy, where I completed the Masters degree in Digital Environment Design at the Nuova Accademia di Belli Arti (NABA), here I met Andrew and TouchDesigner. During the Master, I’ve planned and carried out numerous installations, performances and interactive clothes. Now I’m working at different studios as Technology or Art Director for installations and events, as well as doing my own personal art projects.
D: Tell us about some of the motivations and concepts driving what you do.
CT: I love working in art installations, specially performances with visuals and interactions. I love working in theatre and bringing new forms of expressions with technology.
D: What was your experience learning TouchDesigner?
CT: Learning TouchDesigner I've experienced difficulties only at the very begining. The whole node thing was a little complicated for me, because I couldn’t get them to work together haha. But it was just a matter of time till I got used to it. I’m no pro, but i manage quite well.
I was also able to get a lot of help from Derivative. Every time I had a question you've answered really quickly, that’s very useful. I’ve found that working with Houdini helped me a lot in the understanding of the nodes.
D: If and how might TouchDesigner differ from the tools you normally use?
CT: It’s so much versatile. You can use it for visuals, but also for interaction, you can work with sound, all in one... and all of it in realtime, I found it very interesting and different. You can see right away the result.
D: With your experience now, how else do you see using TouchDesigner? Are there things you see being able to do with it? Has it opened any doors?
CT: TouchDesigner invites exploration, I’m still surprising myself with the things I discover everytime I work with it! It has opened many doors for me. I’ve connected Arduinos, sensors, and its always works, and you find amazing things to do with it. It just takes imagination and TouchDesigner software! Imagination, TouchDesigner and go! If you have any questions then the forum its really helpful.
D: What next? What are your future plans?
CT: My plans for the future would be to continue working in art installations and performances here in Argentina and other places. And to use TouchDesigner always. Whenever I can!
NIMA GAZESTANI, Interaction Designer + Teaching Assistant NABA
I graduated in Product Design at the Faculty of Architecture in Genoa, I worked as a 3D modeler for different studios of architecture; then i started a master in "Digital Environment Design" in NABA - Milan.
Most used tools: creativity and drawing (first place), 3D Studio Max, Photoshop, Illustrator, Premiere, TouchDesigner, Arduino, Mocolo, CCV, TouchOsc (ipad), WiiMote and any other devices (waiting to buy a Kinect).
My first project with TouchDesigner
I felt in love immaediately with TouchDesigner and I started to use it for my first student group project: Hyperwish - Move Around Desires. This was one of my favourites project, it included: multiprojection, motion tracking, Arduino, Processing, Max/MSP, MySQL database, a web-site connected to the installation.
I designed (with the help of Andrew Quinn) a tunnel made with entire desires floating and constellations of words; the words were taken from a web-page, automatically updated every 30 seconds and they were divided in primary words (inserted by users) and a semantic research of hyponyms of the unique words written by people. You could navigate inside the "tunnel of desires" using a light-sphere, tracked from top (using Mocolo, developed by one of our professor).
Hyperwish had always a great success, we presented it at:
- NABA
- Biennal of Sculpture of Carrara
- Triennale Bovisa - Milan (during Salone del Mobile)
- Biennal of young artist in Skopje
- Fabbrica del Vapore - Milan
D: You appear to be a very passionate person Nima - what motivates and excites you?
Nima: I love to do simple and easily understandable interactive installations and interfacing with other people working in the same field but with different skills. I LOVE sound-reactive visuals! I think it's the most beautiful thing in the world! I love mapping a building or a sculpture with a projection, I love to mix human natural gestures and technology. I love learning and being inspired by others: designers, dancers, singers, lovers, students, kids, books, movies, urbanism etc..
D: You have been helping to teach TouchDesigner for a few years. What has the process been? How do you teach it?
NG: I usually started teaching the basic of TouchDesigner, I remember very well how I learned it and I try to show to the student the same process, using simple nodes, starting from compositing, using the mouse panel and audio stream to make them interactive.
I really appreciate when they learn few things, but deeply. I know that at the beginning if you use too many nodes you get confused, and the software takes you on a different way from your thoughts.
D: Good advice, start small.
NG: Yes. An example of a simple and cool installation that you can do with 3 nodes it comes from a book of Dan Graham: if you place a person between a mirror and a monitor and you film him from behind the monitor and you stream the movie with the delay of 4sec in the screen, you'll have a beautiful effect of recursion :) i attached a quick-draw!
NG: TouchDesigner was used in every project and it communicate greatly with other software via OSC.
D: How does TouchDesigner differ from other tools you use?
NG: One of the biggest difference from other tools is that it has a very friendly user interface. Another quality is that is the most complete realtime 3D and compositing software, including FBX, and all the communication protocol that allows you to add new tools and make them communicate with TD.
D: With your experience now, how else do you see using TouchDesigner? Are there things you see being able to do with it? Has it opened any doors?
NG: For now I just did few projects as a freelance, but I'm very excited to always know how to realize an interface or an environment, without becoming crazy!
I always thought that an application for TouchDesigner in the future could be for architecture studios, to show to clients projects and environment projected on 3 screens or on a 360 degree screen, so clients can move inside the virtual building and have a point of view and dimension as could be in reality.
D: What next? What are your future plans? Any plans that include using TouchDesigner?
NG: My dream is to become a pro with TouchDesigner and be able to work as a freelance everywhere. I would love to come to Canada next year to work and learn at Derivative and then...
All my plans includes Touch Designer!! :)
ELENA CASTELLINI, Set + Interaction Designer
Elena: I attended the set design course at the "Brera Belle Arti" academy and after that the Master course in Digital Exhibit and Set Design at NABA academy in Milan. During this master course I've attended the course of TouchDesigner of Andrew Quinn and Nima Gazestani. Mostly it was a practical course; the first part of the lessons was based on the analysis and explanation of examples of projects under the teacher's supervision, trying to understand the "Touch" parameters functionallity. The other part of the course was related to the implementation of new projects.
During this period some students carried out digital scenographies for a contemporary musical exhibition at Milan's "dal verme" theater. It was a wonderful experience, a deeply emotional adventure full of personal satisfaction.
During the exhibition I took care of two projects, the track "Halak" of the musician Christian Cassinelli and "Aphrodite" of Ruggero Laganà. The scenographies were projected on a 7x2 meters area behind the orchestra. According to the Jewish meaning of "halak" which is walk, we decided to respect the soul of the track, a sort of spiritual trip, a pilgrimage starting from a town, through nature, till arriving to a cathedral.
The track started with coloured fog which slowly was dispelled leaving the space to three big sliding images projected on the whole area of projection. At the same time there were thin blue and white vertical sliding lines which increased or reduced their number depending on the sound intensity. Sometimes, instead of being automatically controlled according to the music, they were directly managed by a "real time" slider. The white lines represented the pilgrims during their journey, when the sound was strongly increased to cover the whole area. We used the z-axis to give depth to the pilgrims who came at times to cover the entire area by hiding the underlying landscape and making the scenery much more abstract.
The passage of Ruggero Lagana "Aphrodite" began with an image composed of a grainy circle in the middle that went from white to yellow, which had a golden effect, fading into the background after a few minutes leaving the scene seven beams of light. Inside of these beams were videos of textures with geometric and natural evolving shapes which were changed on the basis of the performance of the piece with variations of color transparency, strength and speed.
Before the master course I had never used TouchDesigner, only graphics software, I found myself in trouble in the beginning to understand the workings of the operators and to adapt the interface that was entirely new to me.The course and video tutorials were very helpful and explained well to learn what kinds of possibilities offers TouchDesigner. What helped me most of all has been studying and learning about the patch for the Halak passage which at first seemed very complicated for my beginner-level knowledge of TouchDesigner.
Another important factor was to experiment alone trying to do every time a new little project, setting myself always a different target. Once you take it into your hand TouchDesigner becomes like a game, the results you get are always interesting and satisfactory, urging you to move forward to experience something new, fresh ideas that grow in tandem with the level of learning.
One nice thing about TouchDesigner is that often with trial and error you get a result but at the same time something unexpected and interesting that can be useful for another project. It is an extremely versatile tool with which you can do such diverse projects, being compatible with other tools opens up enormous possibilities.
I tried to use other similar software, but this is the interface with which I'm most happy especially because you can see every step. Thanks to the graphical representation within each operator it is easy to see the errors, look at all the steps which is quite useful especially for those who are starting at the beginning and coming from the visual arts.
After NABA I used TouchDesigner to do a video mapping, I had prepared a tent for a 3D projection that would be used as a facade for the countdown of the opening of a building. It moved as if it had been moved by the wind, the effect was very nice in videos, but the street was brightly lit so we could not use it. Perhaps soon I will have to prepare a video projection on a screen but this long and narrow but this project yet to be decided for the license and resolution of TouchDesigner.
I would like to continue to develop projects to create scenes with TouchDesigner, and digital interactive installations.