Company Post

Elsewhere Sound Space: Twitch's First Space Cult

With the live music scene but a dim pre-pandemic memory, Bushwick nightclub and music venue Elsewhere  known for its prescience in booking next-gen talent and shaping music scenes, launches “Elsewhere Sound Space” a monthly virtual live music/comedy/variety show streaming on Twitch that promises a quirky and next-level interactive experience. The show follows the sci-fi misadventures of a music deity marooned in space, and their curious cultural dispatches back to Earth. Broadcasting live from the Astral Spa & Sound Bath, host Peter Smith is on a chilled-out mission to bring the joy and community of live music to Earthlings, who are in their most quarantined and isolated state.  
TouchDesigner is the meat, potatoes, glue, scaffolding, butterscotch, etc., of this show - it's our medium.
Hard Work Party chief Noah Norman who epically programmed the entire show application tells us about this "growing TouchDeisgner beast" and his exploits over the last year in building a truly interactive totally-custom new kind of show that would not have been possible to do in any other program. "I can say that attempting to do this kind of thing with hardware would have come with a budget orders of magnitude greater than we had to spend and would have required a small army to be onsite during the shoot, something entirely impractical during COVID," he says. "TouchDesigner is the meat, potatoes, glue, scaffolding, butterscotch, etc., of this show - it's our medium.”
See the full inSession interview with Noah Norman and a deep dive into the inner workings of the Elsewhere Twitch broadcasts - interview by Derivative founder Greg Hermanovic with hosts Markus Heckmannn and Isabelle Rousset.

Also check out Noah's deeper dive into the beautifully-designed python systems developed for Elsewhere on InteractiveImmersive IO.    And see the first episode of Elsewhere with Princess Nokia.
Elsewhere Presents...An Astral Talk Show & Live Music Transmission
with elsewhere shuttered and the world in a quarantined state, we spent 2020 scanning the skies for signs of cultural life. we have an exciting discovery to share.
later this month we’ll be broadcasting the inaugural transmission of Elsewhere Sound Space - a signal we’ve been pirating with a coathanger and an iPhone 6 - an invigorating and perplexing missive emanating from a space vessel calling itself an “astral spa & sound bath” (???)
join us as we follow the artist conversations, live performances, and intergalactic comedic misadventures of a celestial music deity and her musical guests marooned in space.
according to our calculations, we anticipate her next cultural tidings of love to NYC will begin live-streaming Jan 28th, at 7:00 PM EST (Earth Standard Time), with astral royalty rapper Princess Nokia
Derivative: Noah what is Elsewhere Sound Space?
Noah Norman: Elsewhere Sound Space is an astral talk show and live music transmission. It’s a comedy variety show and music performance. It’s an attempt to create something new in a new medium - a completely interactive live-streamed show where viewers’ participation is more than just discussing with one another in the chat.
Every episode, when the relay moon aligns, the exile spa vessel Elsewhere Sound Space is able to transmit and receive to the enlightened faithful on Earth who wish to receive their emanations, and a special guest comes aboard to entertain our glorious leader Peter with live performances.
The show is fully Twitch-native, and users can reach through the screen and poke host Peter Smith, her guests, and manipulate the ship and the feed itself.

Derivative: Who instigated the show and how did it come about?

Noah Norman: Venues and musicians are struggling. The events business has been all but dead for months. NYC’s live music scene is hanging on by a thread. But the community around Elsewhere and the emerging artists they champion are as hungry as ever for connection, for new experiences, and for live music.

With that is mind the production team - Elsewhere, Peter, Dark Igloo and myself - stepped back to consider what about the experience of seeing music live could be brought into a ‘virtual’ or fully-digital experience at home and what about various ways of doing that brought out new opportunities. We wanted to make sure we were creating something that played to the strengths of our chosen medium and that wasn’t just an attempt at creating a facsimile of the experience of being in-person at a venue, however realistic or stylized.

We wanted to create something entirely new, and to take advantage of the opportunity presented by the lockdowns - the uniquely high tolerance for screen time and the attention to detail that comes from being deprived of novel experiences and spectacle - to help transport viewers somewhere else with the help of their participation.

We think Twitch represents a new medium in this regard - it’s a platform that truly enables bidirectional communication and one where the conversation and activities of the viewers aren’t just a sideshow but are really the point of what’s happening. That’s a great fit for Elsewhere, because Elsewhere has always been, and continues to be, a community.

Derivative: Can you take us through the staggering system you built and explain why all the development is in TouchDesigner?

Our show application is a growing TouchDesigner beast. It’s a video pipeline with an A/B switcher as its dominant metaphor, but that core component obscures a lot of complexity and processing.

At the top of the pipeline are two Notch TOPs that do our chroma keying and use NVidia’s AR SDK to do face-tracking and sling that telemetry out over OSC back into the Touch context so we can punch in on faces to vary the composition of our shots.

From there every scene in the A/B engine is its own modular compositor with its own set of rules. Some are simply flat compositing of sources over backgrounds, but some are 3D, some are in a weird middle, and all are composed using colorful, surreal original graphics and animations made by our friends at Dark Igloo and Felt.

On top of the textures composed by the main switcher/scene composer pipeline are a number of augments and ‘gags’ connected to the Twitch API. There are screen-space effects, interactive simulations, a particle system disguised as a ticker, a voting engine, network control of an Ableton Live instance, up to 7 cameras at once, live video calling, multiple record decks, and control of physical gags within the studio, all driven by the main TouchDesigner process.

Our connection to Twitch listens for PubSub events like Bits, Channel Points, and Subscribers, and we've made custom Channel Points events for the show. We hit the Twitch API for stats that show up in places like the ticker and the credits, and to inform how we level-set on things that rely on certain amounts of participation like voting and group bits goals. Lastly, we're of course connected to the Twitch chat so we can listen for '!' commands (which are not a built-in Twitch thing but are a convention among interactive channels) and so we can reply as Elsewhere in the chat from Touch. Twitch handling is in its own Touch process and pipes back and forth all the data and events via strings over OSC.

In the studio itself, that same TouchDesigner instance presents the operator (me) two screens, a UHD screen to our host Peter, which is packed with heads-up info and video feeds as well as a Web Browser COMP render of the Twitch chat, and four screens in the guests’ space, each of which presents varying views on what’s happening in the show, messaging from backstage, and prompts from the viewers.

Peter and I each have a Stream Deck controller - a little grid of buttons with LCD screens behind them, that I’m able to rapidly modify to include new ideas as we come up with them. That’s our cheapo control deck and it’s way better than control surfaces costing ten times as much because the backlit screens make them easy to read and the software makes them easy to rearrange on the fly.

Having done quite a few shows that aspired to this level of juicy digital interactivity and aliveness, I can say that attempting to do this kind of thing with hardware would have come with a budget orders of magnitude greater than we had to spend and would have required a small army to be onsite during the shoot, something entirely impractical during COVID.

Trying to do this entirely in software would have required either making an ongoing and increasingly complex series of crazymaking compromises and brittle connections between off-the-shelf softwares or far more time than we had to do it in something less nimble than TouchDesigner - something like oFx or Cinder.

I’m the only developer on the project. We’re using one machine. We have one Touch license. And on-set we’ll have a COVID-friendly production of one person per room in five rooms.

That’s really the power of TouchDesigner to me. It’s in this powerful sweet spot of abstraction - I can put together enormously complex video pipelines in a stream-of-consciousness front-to-back method without it being a startups’ worth of engineering effort. I can bolt on to that pipeline interactivity, network connections, physical interfaces, and processing in profound and entirely custom ways. I can make user interfaces and change them quickly. I can go from laughing about something in a production meeting to having it implemented in two hours.

It’s been refreshing to show this to the team on this job. We’re all old friends but they’ve never gotten to see what TouchDesigner is really capable of in realtime, so I’ve been able to surprise them over and over again by bringing to life the craziest stuff we can collectively come up with and by repeatedly saying ‘the answer is almost always yes.’

Follow Elsewhere's Twitch channel to be notified when Elsewhere Sound Space streams monthly. Upcoming shows include Starchild and the New Romantic on 2/23 at 7pm ET, and Paperboy Prince on 3/23 at 7pm ET.



Created by: Elsewhere Studios

Hosted by: Peter Smith

Technical Direction: Noah Norman (Hard Work Party)

Art Direction: Dark Igloo

Creative Direction: Jake Rosenthal (Elsewhere) & Noah Norman

Produced by: Jake Rosenthal (Elsewhere)

Creative Writing: Ashok "Dap" Kondabolu, Dark Igloo, Peter Smith

Studio Direction: Chris Madden

Sound Design: Chris Madden

Live Editing: Noah Norman

Talent Booking: Rami Haykal (Elsewhere)

Video Art: Devon Moore (FELT Zine)

Production Consult: Chris Willmore

Special Thanks: Will Adams, Meredith Suzuki