Kinect Floor Track 088

Module for tracking multiple people on a clear, flat floor space using a Kinect v2. Recommended position for the Kinect is approximately 14-15 feet (~4.5 meters) above the floor, pointed directly down, perpendicular to the floor plane.

Currently all data is in the Kinect depth image pixelspace which is 512x424. I plan to add a normalized output option.

I’ve been running this module as its own instance, usually locally. Data is sent to a local Touch instance via shared memory and to a networked Touch instance via IP.

Figured I’d take my first crack at sharing something as many of you are so generous. Hope it’s of use! Please let me know if you have any feedback.

Related post → facebook.com/groups/touchde … 185179685/

UPDATE:
Latest version of KinectFloorTrack attached to this original post. This module works with TouchDesigner 088 but not 099.
KinectFloorTrack_088.3.tox (17.7 KB)

Really nice tool, wish I had a kinect on me to test it out. I went through it and found some areas that could be optimized, left my notes below:

  1. in KinectFloorTrack/left/health/vram
    There’s a lot of time spent cooking all the TOPs. A way to really optimize that. For the Text TOPs because we know that gpu mem free won’t change, you can just use an Execute DAT that runs on system startup that reads that value once, and writes it once. Then that ‘free’ TOP never cooks again. For the ‘used’ you can use a similar trick to control needless cooks. I’d connect the gpu_mem_used to a math chop and round it (faster than rounding in Python), then use a CHOP Execute DAT on that and have it only update the ‘used’ TOPs value if it changes. Currently it basically forces a cook every frame. Same thing for the ‘bar’. Have to stop it from cooking cause it’s very slow. A trick is to cache all the frames of the bars possible positions into a cache TOP then use a Cache select TOP and use your ‘math1’ values as the frame index. Incredibly fast and a trick from someone else’s UI took kit.

  2. In both the KinectFloorTrack/left/health/ comps they have their own Perform CHOP. Use one and select chops.

  3. Same as 1) but for left/info/num_detected/value. It updates every frame, when you could probably just have it update when the number of blobs changes.

  4. In general all your UI elements have Python based resolutions, which is nice and convenient, but basically doubles the cook time of any UI element that needs to be recooked. Something to think about removing.

  5. left/info/avg_speed/value, you’re better off using your mod in the ‘Text’ parameter, as it won’t error out on None value, where as using the ‘Value’ appending will error out if for some reason (like not having a kinect) causes null channels.

  6. General comment for most of your UI, you could greatly speed it up by pushing value changes instead of pulling them, like I said in 1)

  7. Personally, I think an on/off switch for the Summary sections grid preview would be useful. Once you set it you probably will never look at it again. Maybe even a full shutdown switch for the whole UI for permanent installs.

  8. I don’t have a kinect so I can’t really see the rest in action, but also probably a shutdown switch on all viz elements would be nice to have, like all the renders, then just have the CHOP/DAT being output.

  9. You may consider having a Kinect comp which could optionally take pre-recorded kinect footage as the input, would make it easier when testing for long periods of time. Then if you upload some test data, I can look at the rest and give input.

  10. Not sure if you considered down-resing to even lower than 512x424, but I recently did some CV work and noticed that having higher resolution doesn’t really offer benefits in most situations, just takes longer to compute, could even down-res, blur it, then feed it in into the blob track.

Hope that helps! If you can do 9) and post some test recording as well, I can keep going.

Thanks for sharing, sounds like it’ll be useful for a lot of folks.

Elburz, thanks so much for taking the time to look at this and giving such thoughtful and insightful feedback. The UI suggestions are super helpful, and I’ll be certainly keeping them in mind as I approach future projects.

I’ve implemented suggestions 1-3 and 5-8 in this updated version. Regarding a shutdown on all viz elements, I am also considering reducing any render TOPs to 1x1 resolution to free up any additional VRAM when the UI is not in use.

  1. I wasn’t sure if the performance gains would be felt beyond setup/configuration, but I’m trying to get into the habit of best practices. Python was certainly helping when prototyping; would you recommend switching to CHOP Export?

  2. I’ve been curious about how to implement a Kinect footage recorder in Touch, especially for the depth buffer, and will look into this more intentionally once I get on the other side of a current project. I’ve been using Kinect Studio in the meantime. I wouldn’t feel comfortable sharing the current data (.XEF file) I’ve been using as I haven’t requested consent from any of the test participants to post it publicly. However, I’ll have this configuration set up again in the next week or two and will certainly share some then!

  3. I tried doing some initial down-res attempts per your suggestion, and then I remembered all of the blobtrack data is in the pixel space which messed with the tag overlay. I’ll work on normalizing the data and will post an update once I have something.

Lastly, I’m considering removing the whole velocity calculation from the KinectFloorTrack/data section. I feel this may be better implemented outside tool (and probably a lot more efficiently than I currently have it).

+1 for exports
python makes for fast prototyping, but is slow for things beyond that phase

I’ve had good luck with using parameter executes to set ui parameters are create or start. You can fill in an export table when a parent parameter changes.

Thanks for your feedback, y’all!

I’ve just uploaded a new version (KinectFloorTrack.3.tox) to the original post above.

All the UI has been changed so it pushes values via CHOP Export instead of Python script.
Matthew, I’m not sure I understand the parameter execute/export table method. Would you be able point me to any examples for me to study?

I added the option to half the blobtrack TOP’s input resolution from 512x424 to 256x212 per Elburz’s suggestion. This dramatically increases speed while giving comparable accuracy. I did notice some degradation in accuracy when many people (20+) are in very close proximity. My next step is to experiment with blur amounts to see if this improves half-res accuracy.

I also removed the velocity/speed calculation and tag sections. My implementation was too slow and is likely better implemented outside the scope of this tool anyways.

Hey Carlos,

This particular approach is pretty brute force, with a bit of better planning this could be done with just a few loops. The idea here is to watch the parent component for changes in the height and width, and then to set the contents of an export table with the appropriate values.

This does all of the math in python - which isn’t as fast as CHOPs, but does remove python from any expressions in UI elements.

To see this thing in action, change the size of container1, and all of the sub components will change as well.

base_changing_contaniers_with_executes.tox (2.12 KB)

That’s a really cool example, Matthew. I hadn’t realized the power of being able to build export tables like that. Looking forward to experimenting and applying this technique.

Thank you!

Glad it makes sense!

My general rule these days is that if the Operator cooks frequently, I use exports. If it cooks infrequently or only at start I’ll use expressions. You can sometimes get the best of both words by using eval DATs along with a substitute to create an export table that updates only during changes. This kind of approach can be useful for Math CHOPs where you need to change the to or from ranges - you don’t want an expression evaluating in the CHOP, but you might still need the flexibility for things to change on the fly.

Another consideration is if you want UI elements that grow or shrink dynamically. Rather than running a script on change you could use a parameter CHOP, and then filter or lag.

Those CHOPs can be expensive if you aren’t careful, so you just need to make sure that you add a null that’s set to selective after them - this should prevent unwanted cooking.