Palette:camSchnappr
Summary
camSchnappr is a interactive mapping application entirely inspired by MAPAMOK, created by Kyle McDonald at the YCAM Interlab. With Kyle publishing the source code for MAPAMOK, we had a chance to look at it and convert it into TouchDesigner. For the original tool and source have a look here:
Kyle describes the significance of the tool as the use of OpenCV's cameraCalibrate to calibrate a projector via a model of the to be mapped structure instead of using a checkerboard. With TouchDesigner’s integration of Python, we were able to implement the same functionality into a standalone tool which lets you map complex structures in a small amount of time.
See also Projection Mapping, Palette:kantanMapper, Vioso, Scalable Display TOP, projectorBlend
What it does
If you have a physical 3D structure plus a virtual 3D model of that structure, you will be able to project a rendered virtual model onto that physical structure perfectly-aligned. You will be selecting points on your 3D Model and align them with the real world positions as you see them on the projector output. In order to support multiple projectors, the output of blend-masks from multiple camSchnapprs is now supported.
Using your 6 or more alignment points, CamSchnappr will compute the position, rotation, scale and viewing angle of a specified TouchDesigner Camera component used for the rendering. Internally, an openCV function called calibrateCamera
will be run and the “intrinsics and extrinsics” of the camera will be calculated and stored in the Camera component.
What you need
- TouchDesigner
- a projector. Note: Make sure your projector has all digital transformations like keystone or zoom reset.
- a physical structure to project on
- a 3D Model of the to-be-mapped structure
Getting Started
camSchnappr is located in the Palette in 'Mapping' folder. To begin, open the Palette and drag camSchnappr
from Palette>Mapping onto your network pane.
Parameters - camSchnappr Page
Project
- Specify a Camera COMP that will be used as the camera to be calibrated. By default the internal Camera COMP called "project" is being used. When dropping a new Camera COMP onto this parameter, a selection of parameters will be created.
Geosop
- Assign the SOP which holds the geometry you are calibrating. Do not specify a SOP that is a Material SOP or is from a branch with a Material SOP in it as this would break the blending. Instead reference the Texture via the Color Map Parameter.
Usepointgroup
- Enable if the source geometry contains point groups that should be used as calibration points. This is useful when dealing with large models where you can't select points but want to use a predetermined set of points.
Geogroup
- The available point groups that can be selected as a source for predetermined calibration points.
Calibrationerror
- Displays the calibration Error returned from OpenCV after cv2.calibrateCamera has been executed. That is, the total sum of squared distances between the observed feature points imagePoints and the projected (using the current estimates for camera parameters and the poses) object points objectPoints.
Openmain
- Open the camSchnappr editing window.
Closemain
- Close the camSchnappr editing window.
Monitor
- Assign which Monitor the output window will open on. This should be your projector.
Outputcanvas
- Enable if the projector output is part of a canvas. The main output might be a single window which is split to multiple projectors.
Projectorres
- The resolution of the individual projector in the output canvas.
Gridpos
- The pixel position of the individual projector in the output canvas from bottom left.
Open
- Open the output window.
Close
- Close the output window.
Openmainedit
- Open the output as a framed window on the main monitor. This can be useful to select and move points on the projection when they are not easily identifiable.
Guides
- Show the guides on the output monitor.
Guidescolor
- Control the color of the guides on the output.
Guidesthickness
- Control the thickness of the guides on the output.
Reversevertex
- Reverse both U and V for the Hull.
Colorrandom
- Color each primitive of the geometry by random. This can be useful when schnapping complex objects.
Colormap
- Assign a TOP which is used as texture in the editing viewport and the output.
Texturealpha
- Control the texture alpha.
Wireframe
- Show or hide the geometrie's wireframe.
Wirewidth
- Control the wireframe strength of the geometry in the output window.
Clear
- Clear the calibration data including any selected points.
Blightdimmer
- Control the backlight amount of the output window.
Wirecolor
- ⊞ - Set the wireframe color of the geometry in the output window.
- Wireframe Color
Wirecolorr
-
- Wireframe Color
Wirecolorg
-
- Wireframe Color
Wirecolorb
-
Blightcolor
- ⊞ - Set the background color of the output window.
- Backlight Color
Blightcolorr
-
- Backlight Color
Blightcolorg
-
- Backlight Color
Blightcolorb
-
Altmodifier
- Control the offset of a selected point when moving it with the Alt+Arrow Keys in the output window.
Parameters - Auto Blend Page
The Autoblend feature uses Light COMPs to retrieve overlapping areas by comapring uvs from the lights projection area. Further it uses the Blend, Gamma and Luminance Parameters to adjust the blend region after the principles outlined in Paul Bourke's Paper: http://paulbourke.net/miscellaneous/edgeblend/
Camschnapprcams
- Specify all other camSchnappr Camera COMPs in order to calculate the blend-mask for the projector.
Blend
- Adjust the power of the blend function.
Gammared
- Adjust the gamma for the red channel.
Gammagreen
- Adjust the gamma for the green channel.
Gammablue
- Adjust the gamma for the blue channel.
Luminance
- Adjust the luminance for the blend function.
Parameters - openCV Page
Near
- This control allows you to designate the near clipping plane. Geometry closer from the lens than this distance will not be visible.
Far
- This control allows you to designate the far clipping planes. Geometry further away from the lens than this distance will not be visible.
Fov
- Specify the initial FOV (field of view) estimation for the camera matrix given to the cv2.calibrateCamera
function.
Intrinsic
- The initial cameraMatrix passed to cv2.calibrateCamera
contains valid initial values of focal length and principal point that are optimized further. Otherwise, the principal point is initially set to the image center ( imageSize is used), and focal distances are computed in a least-squares fashion. It sets the flag cv2.CALIB_USE_INTRINSIC_GUESS
Fixaspect
- This considers only the vertical focal length as a free parameter. This should always be enabled unless you have an unusual projector with non-square pixels. It sets the flag cv2.CALIB_FIX_ASPECT_RATIO
Zerotangentdist
- Tangential distortion is set to zero. It's enabled by default because most projectors have very little tangential distortion. It sets the flag cv2.CALIB_ZERO_TANGENT_DIST
Fixprincipal
- The principal point is not changed during the optimization. You should enable this if you have a high quality lens with zero lens shift. It sets the flag cv2.CALIB_FIX_PRINCIPAL_POINT
Fixfocal
-
Fixk1
- The corresponding radial distortion coefficient is not changed during the optimization. For extremely wide fisheye lenses or lenses with radial distortion try enabling these. It sets the flag cv2.CALIB_FIX_K1
Fixk2
- The corresponding radial distortion coefficient is not changed during the optimization. For extremely wide fisheye lenses or lenses with radial distortion try enabling these. It sets the flag cv2.CALIB_FIX_K2
Fixk3
- The corresponding radial distortion coefficient is not changed during the optimization. For extremely wide fisheye lenses or lenses with radial distortion try enabling these. It sets the flag cv2.CALIB_FIX_K3
Maxiterations
- Specify the number of iterations the algorithm should terminate after.
Precision
- Specify the value of accuracy at which, when reaching it, the algorithm should terminate.
Parameters - ArcBall Page
The ArcBall parameters give control over the transform multipliers of the Main Window's arcBall camera. When using geometries with large dimensions, it can be necessary to adjust the Translate, Dolly and Rotate Multipliers to efficiently make use of the ArcBall Viewport.
Transmult
- Control the Translate Multiplier of the ArcBall camera.
Dolmult
- Control the Dolly Multiplier of the Arcball camera.
Rotmult
- Control the Rotate Multiplier of the Arcball camera.
Parameters - OSC Page
camSchnappr now allows you to control it from a mobile device using TouchOSC Download the TouchOSC layouts for IPad File:camSchnappr.touchosc or IPhone File:camSchnapprIPhone.touchosc. Make sure to turn on Touch Messages (/z) in the TouchOSC options as some of the interactions require this message to be send through.
Active
- Activate OSC access to camSchnappr via TouchOSC.
Port
- Set the portnumber specified in TouchOSC here. MAke sure in TouchOSC incoming and outgoing ports are set to the same value.
Parameters - About Page
Help
- Opens this page.
Version
- Current version of this COMP.
Toxsavebuild
- TouchDesigner build this component was saved in.
The Viewport
Open the viewport by clicking the Open Main Window parameter.
You can interact with the viewport as follows:
- NEW***:
- left-mouse click and drag to tumble
- middle-mouse click and drag to zoom
- right-mouse click and drag to pan
- h to home the geometry
- create control points by ctrl+left-mouse clicking on the geometry
- make control points active by ctrl+left-mouse clicking on blue spheres
- delete control points by ctrl+right-mouse clicking onto spheres
- tab to cycle through selected points
When you select points a little flag with the point number will be displayed.
The Mapping Viewport
Once you selected the correct output Monitor via the Output Monitor parameter, click the Open Output parameter.
The interface is quite simple:
- a red crosshair indicates your current cursor position on the screen
- a dark yellow crosshair indicates an inactive point previously selected to be mapped in the main viewport
- a yellow crosshair indicates an active point previously selected to be mapped in the main viewport
You can drag points to their real-world position on the screen or use the arrow keys on the keyboard. For faster movement, use alt+arrowkey to move points.
You can also Shift+Left-click onto the viewport to move points directly to their corresponding real-world position.
Tab will cycle through the available points.
Using the keyboard arrow keys, you can move the points 1 pixel at a time.
Using Alt+Arrow keyboard keys will move the point by the Alt-Multiplier Parameter amount (by default 10 pixel).
Workflow
- Create a simple render setup with the model of the object you want to map onto.
- select the camSchnappr Camera COMP.
- Optional: Drag your Camera COMP onto the Project Parameter of camSchnappr.
- Drag the SOP containing the geometry of the to be mapped object onto the Geo SOP parameter.
- Select the number of the output your Projector is connected to via the Output Monitor parameter.
- Open the Main Window of camSchnappr by clicking the Open Main Window parameter.
- Open the mapping Viewport by clicking the Open Output parameter, if somehow the output now overlays your main viewport, just hit escape and select a different Monitor Output number via the Output Monitor parameter.
- In the main viewport align the geometry to the camera so it’s similar to what the projector sees.
- Select a point in the main viewport and move it in the mapper viewport to the corresponding position on the real world object. You can also hold the Shift key and click in the mapper viewport onto the corresponding real world position.
- Repeat the last step for at least 6 points total.
- After you have aligned 6 points, the camera should be calibrated and you should see the projection mapped onto the object.
- If everything is correct you can close the mapping output and the camSchnappr viewport by clicking the Close Output and Close camSchnappr parameter. The Camera Calibration values are saved inside your camSchnappr Project Camera COMP as 2 Table DATs.
- If you used an externa Camera COMP as a project, you can now also delete camSchnappr.
OSC Controls
camSchnappr can be controlled using following osc channels:
- /1/selectPointNext [0|1] - selects the next controlpoint
- /1/selectPointPrev [0|1] - selects the prev control point
- /1/pointCoarse [float] [float] - move the currently selected controlpoint by u/v amount.
- /1/pointFine [float] [float] - move the currently selected controlpoint by a fraction of u/v amount.
- /1/pointFine/z [0|1] - if
1
start fine movement, if0
finish fine movement - /1/left [0|1] - if
1
move selected point by 1 pixel left. - /1/right [0|1] - if
1
move selected point by 1 pixel right. - /1/up [0|1] - if
1
move selected point by 1 pixel up. - /1/down [0|1] - if
1
move selected point by 1 pixel down. - /1/altleft [0|1] - if
1
move selected point by 10 pixels left. - /1/altright [0|1] - if
1
move selected point by 10 pixels right. - /1/altup [0|1] - if
1
move selected point by 10 pixels up. - /1/altdown [0|1] - if
1
move selected point by 10 pixels down. - /1/openCloseOutput [0|1] - open / close the main poutput.
- /1/selectCamSchnappr [0|1] - select the next camSchnappr in your setup.
- /1/selectCamSchnapprPrev [0|1] - select the previous camSchnappr in your setup.
Operator Outputs
- Output 0 - When using multiple camSchnappr cameras, this output will be the blendmask between them.
TouchDesigner Build: