Layout3D - Run Layout Estimation models locally to build SOP geometry of 360 Panoramas For Beginners

TDLayout3D is a drop-in TouchDesigner .tox that takes a panoramic image (or 360° video) and spits out a full 3D model of the room inside it — corners, walls, floor, ceiling — ready to use for projection mapping, VFX, set design, or just cool-looking renders.

It's the kind of tool that replaces an hour of measuring and modelling with about 30 seconds of dragging a pano into TouchDesigner.

What it does

You feed it a 360° equirectangular panorama. It runs a room-layout AI (HorizonNet) inside TouchDesigner's embedded browser — zero Python dependencies, fully offline, cross-platform. You get back:

• A textured 3D room SOP you can drop straight into a render or use for projection mapping

• A plain (untextured) room SOP for clean geometry

• A corners DAT with every corner's 3D coordinates and pano UVs

• A CamSchnappr-ready DAT (screen-space + world-space correspondence table) for direct import into projection-mapping tools

• A metrics CHOP with floor area, perimeter, volume, wall count, inference time, and confidence

• An overlay TOP showing the detected layout drawn over your pano

• A live-rendered preview TOP — the pano texture projected onto the reconstructed room, semi-transparent

Everything updates in real time as you adjust the layout by hand.

Who it's for

This is most useful for live events and projection mapping artists who need to map a venue fast — snap a 360 photo, drop it in, hand-tune the corners, export the geometry, project. Done.

It's also great for:

• Previz and VFX work where you need a quick 3D stand-in of a location

• Real-estate and architectural visualisation

• Game levels and backgrounds from real-world scans

• Anything where you've got a pano and wish it were a 3D model

The clever bit — the manual lineup tool

The AI is impressive but panoramas are warped, rooms are messy, and sometimes the model picks the edge of a couch instead of the actual wall. So there's a hands-on refinement layer built in, right on top of the detected layout.

• Drag any corner box — green ones are floor corners, magenta are ceiling corners. Each one moves independently, so you can line each one up with the actual corner in your warped pano.

• Drag any wall line — every line has a bend anchor at its midpoint. Pull it anywhere to curve that wall smoothly. Release and it sticks. Click it to reset to straight.

• Click on a wall — drops in a new corner pair (floor + ceiling) wherever you clicked, so you can add wall indentations, closets, alcoves, L-shaped rooms.

• Click on a corner — deletes it.

The 3D geometry stays as a solid boxy room no matter how warped your pano is — you're adjusting the 2D overlay and the TOX does the 3D maths underneath. Even with a dodgy inference or a gnarly fisheye pano, a minute of dragging gets you a clean mappable room.

How it works under the hood

The inference runs in-browser via onnxruntime-web, using TouchDesigner's built-in Chromium renderer and WebSocket server. That means:

• No pip install anything — it's all self-contained in the .tox

• WebGPU accelerated when available, falls back to WebGL/WASM automatically

• 150–400 ms per inference on most modern GPUs

• Works offline, no cloud calls, no subscription

• Cross-platform (macOS + Windows)

The model is HorizonNet (CVPR 2019, Cheng Sun et al., MIT license), shipped with the Structured3D weight variant — trained on synthetic data, which is the cleanest option for commercial use.

Installation

1. Download the zip from the post attachment.

2. Unzip it anywhere on your drive.

3. Open TouchDesigner (2023.12370 or newer recommended).

4. Drag layout3d.tox from the unzipped tox/ folder into a fresh project.

5. Wire a Movie File In TOP into the TOX's input — point it at a 360° equirectangular panorama (2:1 aspect ratio, e.g. 4096×2048).

6. Pulse Settings → Infer Now once.

7. Inspect the outputs — you should have corner boxes on the overlay, a 3D room in the SOP outputs, and everything populated in the DATs and CHOPs.

Total setup time, first try: under a minute.

Input formats

• 360° equirectangular panoramas (primary input — JPG/PNG at 2:1 aspect, e.g. Ricoh Theta, Insta360, PTGui/Lightroom stitches). Native HorizonNet input, best quality.

• 360° video — works frame-by-frame. Use Insta360 in webcam mode or a pre-recorded file. Turn on Auto-Infer for live-updating layouts.

• Stitched multi-photo panoramas — same as single panos, just make sure the final output is 2:1 equirectangular.

One gotcha: phone "panorama mode" JPGs are cylindrical, not equirectangular — they won't work directly. You'd need to convert them first.

Use with CamSchnappr / projection mapping

The camschnappr_out DAT contains a ready-to-use correspondence table: every corner's 3D world position paired with its screen position (in 1024×512 pano pixels). Drop it into CamSchnappr as your point list and you've got a calibrated camera solve from a single photo.

It'll likely work with other projection mapping tools too — but I've only fully tested CamSchnappr. Beta testers welcome.

Cross-platform — beta testers wanted

I've tested this extensively on macOS (Apple Silicon) — runs great, WebGPU, 150–300 ms inference.

I don't have a Windows PC to test on. It should work — WebGPU is available on Windows TD, and the fallback chain (WebGPU → WebGL → WASM) means it'll degrade gracefully — but I'd love Patreon supporters to try it and let me know. Bug reports welcome in the comments.

What's included in the download

• tox/layout3d.tox — the plugin (drop into any TD project)

• examples/sample_panos/ — test panoramas so you can verify it works

• docs/MANUAL.md — one-file full documentation (this post is the summary)

• Full source code + build scripts (MIT-wrapped around the MIT HorizonNet model)

Credits

Built on HorizonNet — Cheng Sun, Chi-Wei Hsiao, Min Sun, Hwann-Tzong Chen, CVPR 2019. Weights via Structured3D (synthetic panos dataset).

Architecture pattern (webRenderTOP + webserverDAT + VFS + ONNX) pioneered by Torin Blankensmith's MediaPipe and YOLO TouchDesigner plugins — this is an independent from-scratch re-implementation of the same concept, no code copied.

TouchDesigner by Derivative.

Support

Drop a comment if you hit anything weird. Include your OS, GPU, and TD version, plus the contents of the log DAT inside the TOX (visible from the Advanced page). I'll iterate fast.

Thanks for supporting the work — this stuff takes months to get right and your pledges are what makes it possible to keep shipping. Enjoy!

V2 of the Layout3D TOX is up. If you grabbed v1, this one is a meaningful upgrade: the TOX now ships a real 3D textured room you can drop straight into a GeometryCOMP, a cross-atlas TOP that properly rectifies each face of the pano, and a bunch of quality-of-life fixes around the click/drag workflow.

What's new

1. Cross-atlas TOP output (the main feature) The final TOP output is now a 4:3 cross unfold of the room:

• Ceiling on top

• Four walls in the middle row

• Floor on the bottom

Each atlas region samples the pano through the correct per-face UV mapping — walls use bezier-curved interpolation across the 24 subdivisions (so bent/tilted panos still look right), floor and ceiling use spherical projection with a corner-snap correction so the interior rectifies from the 360° image while the corners lock exactly onto the wall/floor/ceiling junctions.

2. First SOP output (null6-equivalent) is now the textured geometry In v1 the first SOP output was plain untextured cuboid geometry. V2 replaces that with the full textured room: floor, ceiling, and four walls, all subdivided, with uv, atlas_uv, and pano_uv vertex attributes. Drop it into a GeometryCOMP with any material that samples the atlas TOP and you get a 3D room.

3. Second SOP output carries a material reference The second SOP output now goes through a Material SOP that auto-assigns a constant material with the atlas TOP as its colormap (and cullface=neither so both sides of every face are visible). Any downstream GeometryCOMP picks up the texture automatically — no extra wiring needed.

4. New room3d_preview renderTOP inside the TOX A new internal render that shows the 3D cuboid with the atlas texture mapped onto it. Good for sanity-checking the output without setting up your own render chain.

5. Unrestricted corner deletion V1's JS runtime refused to delete corners when the polygon wasn't "Manhattan-aligned" — in practice that meant any corner you added couldn't be removed without first undoing every other edit. V2 loosens this: click any corner and it's gone, as long as the polygon stays above the 4-wall minimum (so the default cuboid is still safe). The patch is applied inside the bundled runtime.js in the VFS — no rebuild needed.

6. New pulses on the Settings tab

• Reset — hard-reloads the JS runtime and re-runs inference to return to the default 4-wall cuboid. Wipes any corner edits or bends you've made.

• Fix Black Preview — if the webrender ever hangs, one pulse reloads the runtime and retriggers inference. Saves you having to restart TD.

7. Portable TOX paths All the internal webserver/tensor-sender/panelmouse callbacks now use parent()-relative references instead of hardcoded /project1/layout3d/... paths. Drop the TOX anywhere in any project — it just works.

Preserved from v1

The inference pipeline (HorizonNet via ONNX Runtime Web), the overlay drag canvas, the panelmouse click/drag, logging, corners/metrics/camschnappr DATs, per-wall bezier bend handling — all identical to v1. If your v1 setup worked, v2's drop-in replaces it without breakage.

Files

• tox/layout3d_v2.tox — the TOX itself (~297 MB, includes the ONNX model in VFS)

• the full zip can be found in the v1 post here

Known caveats

• The atlas cross uses a 4:3 layout assuming a cuboid room. Non-Manhattan rooms still work but the unfold looks less clean.

• On first load the webserver auto-picks a free port via firstrun. If the preview is ever black on load, pulse Fix Black Preview.

Thanks as always for supporting the project — v3 ideas on the board include a proper GLSL per-face perspective rectifier (killing the spherical singularity entirely), USDZ export, and a room walkthrough camera.

Drop feedback in the comments if you take it for a spin.