Login | Register 
FEATURES
APPLICATIONS
DOWNLOADS
EDUCATION
BLOG
WIKI
FORUM
STORE

Kinect2 - Projector Calibration

Share your components here. First one to 100 wins!

Kinect2 - Projector Calibration

Postby snaut » Wed Oct 03, 2018 1:11 pm

Admittingly a bit late to the party but hopefully helpful anyways, I finally finished this years old project and packed it into a component. With being able to read TOPs directly into numpy arrays this now is a pretty clean solution:
A component which let's you calibrate the projector to your kinect so that you can reproject the skeleton or pointcloud onto the physical scene you are looking at with your kinect camera.

In essence I'm following step by step what Elliot Woods did here https://www.youtube.com/watch?v=llQM-OGsETQ (7 years ago) :lol:

What do you need:
  • a Kinect2
  • a projector
  • a board of some kind that you can project onto (foam core or cardboard)
Step by Step instructions:
phpBB [video]

  • place the kinect2 and your projector
  • on the Kinect Projector Calibration Parameter Page click "Open" to see the control screen
  • select the output Monitor which should be your projector
  • check that the Projector Resolution in the parameters matches the actual projector resolution
  • Click the "Open Checkerboard" parameter to project a checkerboard
  • use a piece of cardboard or similar and make sure the checkboard lands on it. You can adjust the size, position and brightness of the grid - double check that the color camera from the kinect can see the grid
  • click "Get Point Pair" to collect the checkerboard corners
  • repeat with different poses of the board for a few times (try 5-10 different positions)
  • If the parameter "Pointpairs Collected" has a sufficient number, click "Calibrate"
  • If everything ran ok, click "Open Pointcloud" to reproject the point cloud onto your scene.
  • You can also turn off the pointcloud and instead project the Skeleton onto somebody entering the scene

The whole thing acts as a camera, so you should be able to just place it into your scene and you can now render from the viewpoint of your projector in relative position of your kinect camera.

The background is quite simple:
In general it's the same Idea as camSchnappr: give the openCV cameraCalibrate a collection of point pairs: 2D points in the screen space and 3D points in the world space.
The 2D points are collected from the projected checkboard itself while the 3D points are collected from the Kinect pointcloud. To get those 3D points, we run the openCV findChessboardCorners function on the color camera and then use the returned uv positions and lookup the point in the pointcloud.
In the component itself, all functionality is contained in the Calibrate Extension.

It worked well in the cases I have tested here... Let me know what should be improved.

cheers
Markus

Version 0.5 posted fixing an issue for Non-Commercial Licenses related to reduced resolution of the Kinect Camera.

Version 0.6 posted fixing an issue for build 099.2018.27550 and greater where dumping the resulting matrix into a table had rows and columns switched.
Attachments
kinectCalibration.tox
Version3 for TouchDesigner 099.2018.27550
(16.91 KiB) Downloaded 103 times
User avatar
snaut
Staff
 
Posts: 865
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: Kinect2 - Projector Calibration

Postby knut » Thu Oct 04, 2018 6:24 am

Thnx for this.
I am very interested in trying it out. However:

1. If I load the tox I get a version conflict warning. I use 2018.25850 and it tells me
I should use 2018.26390. I cant find that version, also not in the experimental builds
2. If I load the tox into my version, the kinectCalibration chop shows an error (no idea
how to copy the text into this mail)
3. If I run the calibration anyhow, I can execute the first steps (open the checkerboard, scale it).
But when hitting the Get Point Pair Button, TD crashes
4. Dont know how to change position of the grid. Scale and Brightness ok.

greetings knut
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Re: Kinect2 - Projector Calibration

Postby snaut » Thu Oct 04, 2018 11:36 am

Hi Knut,

sorry for the version conflict, but we just released a new build to download today.
What build were you trying this with and what was the error that the Kinect CHOP was showing (doesn't have to be exactly the error - just to get an idea of where it might have come from)

cheers
Markus
User avatar
snaut
Staff
 
Posts: 865
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: Kinect2 - Projector Calibration

Postby knut » Thu Oct 04, 2018 3:03 pm

hi markus
the build I used was 25850, the newest from td 99.
I am away from my machine now..so details on
the error text tomorrow..sorry
greetings knut
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Re: Kinect2 - Projector Calibration

Postby knut » Fri Oct 05, 2018 6:48 am

Hey Markus
1. This morning with TD 2018.25850 and your .tox the chop-error message disappeared. Strange.
I had my PC switched off over night...the only thing that I changed (fingers crossed)
Rest is unchanged: After hitting get-point-pair TD closes. I cant find a crash file...
2. After that I installed 2018.26450.
No errors from the chop but same behaeviuor: TD silently closes after hitting get point pair
button (of course I did the other steps first: Projected checkboard an white surface)

I am on windows 10
GPU GTX 1080, Driver: 397.93

greetings knut
Attachments
20181005_124210.jpg
20181005_124210.jpg (235.78 KiB) Viewed 3191 times
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Re: Kinect2 - Projector Calibration

Postby knut » Fri Oct 05, 2018 11:08 am

..and markus
please let me know what I can do in addition.
I am very much interested in getting this going in
my environment.
knut
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Re: Kinect2 - Projector Calibration

Postby snaut » Fri Oct 05, 2018 12:56 pm

Hi Knut,

and I suspect no dmp file created?
Maybe we can trace the error with this script:

Code: Select all
import numpy as np
import cv2

# capture a TOP into a numpyArray
img = op('camColor').numpyArray()*255

# remove the alpha channel
img = img[:,:,:3]

# convert to 8 bit
img = img.astype(np.uint8)

# convert to gray scale
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

# find corners
ret, corners = cv2.findChessboardCorners(gray, (gridResX, gridResY), None)


If this closes TouchDesigner, we should comment out lines starting from the bottom.
If it runs we can go further and check if the cornerSubPix or calibrateCamera functions are causing this behavior.

Best
Markus
User avatar
snaut
Staff
 
Posts: 865
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: Kinect2 - Projector Calibration

Postby knut » Fri Oct 05, 2018 1:27 pm

so you want me to put this in a text dat and then execute
that text dat..correct?

no dmp file created..I searched for touchdesigner99*
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Re: Kinect2 - Projector Calibration

Postby snaut » Fri Oct 05, 2018 3:01 pm

yup, if you could create a TOP (can be a constant TOP) and call it camColor and then paste the script into a Text DAT and run it.
User avatar
snaut
Staff
 
Posts: 865
Joined: Mon Nov 12, 2007 6:21 pm
Location: Toronto

Re: Kinect2 - Projector Calibration

Postby knut » Fri Oct 05, 2018 3:09 pm

any chance of a more interactive session within the next hour?
it's a 30 min drive for me back to the office.
otherwise I would do it tomorrow morning.
knut
 
Posts: 20
Joined: Sun Apr 19, 2015 9:12 am

Next

Return to Shared .tox Components

Who is online

Users browsing this forum: No registered users and 1 guest