ADDED: opencv v4

opencv.org/opencv-4-0-0-beta.html

hey barakooda,

whats your approach in using opencv within TD? I recently used opencv’s face detection on photographs and I’m reading the TOP as numpyarray in python scripts, like its done in the Kinect Calibration extension.

import cv2
import numpy as np

def onOffToOn(channel, sampleIndex, val, prev):

	img = op('image').numpyArray()*255
	img = img[:,:,:3]
	img = img.astype(np.uint8)
	imgGray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

	cascPath = me.var('TOUCH') + '/haarcascade_frontalface_default.xml'
	# Create the haar cascade
	faceCascade = cv2.CascadeClassifier(cascPath)

	# Detect faces in the image
	detections = faceCascade.detectMultiScale(
	    imgGray,
	    scaleFactor=1.1,
	    minNeighbors=2,
	    minSize=(10, 10),
	    #flags=0
	)

	blabla()

	return

I mean this works, and in this case since it was on loaded images, so the performance didnt matter so much. But if I chose a camera feed for example, the software went down to 5 fps. How do you use the library in TD?

  1. make sure you acquire initialization data only one time
    this code for example should not be coocked every frame.
cascPath = me.var('TOUCH') + '/haarcascade_frontalface_default.xml'

generally, divide your code to what is suppose to be cooked one time and what is needed to
cook every frame.

2.maybe this will work faster and save assignments.

image = ((op('image').numpyArray()[...,0])*255).astype(np.uint8)

3.try to work only with grayscale as input - do as less operation as possible on python\opencv.

4.try to reduce size of texture.

  1. if you will want high performance, you will have to move into c++
    but i saw already someone release example in python to receive texture faster
    thru python wrap of opengl ( if i`m not wrong).

Look on this example -make changes to your case :
not the most efficient but it will might get you closer to reasonable FPS depend on your case).
I`m getting 30+ FPS for face landmark detection.

[code]from imutils import face_utils
import imutils
import dlib
import cv2
import numpy as np

mult = 1/255

width = 1/op(‘image’).width
height = 1/op(‘image’).height

detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(r"C:\Users\barak.koren\Downloads\shape_predictor_68_face_landmarks.dat")

pointsTable = op(‘LandMarkPoints’)

def onPreCook(changeOp):
image = ((op(‘image’).numpyArray()[…,0])*255).astype(np.uint8)
rects = detector(image, 0)

pointsTable.clear()
for rect in rects:
	shape = predictor(image, rect)
	shape = face_utils.shape_to_np(shape)
	
	pointsTable.appendCol(shape[...,0] * width)
	pointsTable.appendCol(1-(shape[...,1] * height))[/code]

I will add to Derivative developers,
if you can add conversion feature from float ( 0-1) to uint8.
I think this code in python makes pipeline slower.:

*255).astype(np.uint8)

if there is function in GL that can convert it before sending - it will be great.

Great. Thanks for your indications, very helpful. I am also trying out dlib now since its apparently much better in detection… My fps are rising :exclamation:

ya i ran into this recently, and would love a way for python opencv to detect the TOP’s pixel format and use that.

However i found that writing out frames (MovieFileOut TOP - Image Sequence), and reading them back (cv2.imread) is faster than numpyarray().

Maybe opencv 4 will change how fast it is, but i doubt it. numpyarray() i think is the major slowdown in this case.

check out here for an example of opencv in c++: viewtopic.php?f=20&t=12586

OpenCV 4.5 has been added as of 2020.42960 and later

1 Like