Microsoft Kinect IR PATTERN PROJECTOR IR CAMERA RGB CAMERA

Microsoft Kinect: How it works?

Microsoft Kinect

Microsoft Kinect

Microsoft Kinect

Microsoft Kinect: Data points3D_Irsys.m: 3xN matrix of 3D points measured and associated RGB color calibData.m : intrinsic and extrinsic parameters of IR and RGB camera imRGB.m : image captured by RGB camera

Microsoft Kinect: Data points3D_IRsys: 3xN matrix of 3D points expressed in IR camera reference system X x1 x… xn Y y1 y… yn Z z1 z… zn Number of points N should be 640x480 = 307200!

Microsoft Kinect: Calibration Data calibData.m : intrinsic and extrinsic parameters of IR and RGB camera K_rgb : 3x3 : intrinsic parameters matrix of RGB camera kc_rgb : 1x5 distortion paramteres of RGB camera K_ir : 3x3 : intrinsic parameters matrix of IR camera kc_ir : 1x5 distortion parameters of IR camera rotRGB2IR : 3x3 rotation matrix from RGB to IR reference system trasRGB2IR : 1x3translation vector of IR reference system respect to RGB

Reminder SC SCCD z x OCCD y OP x p z f y OC Il modello geometrico (ideale) della telecamera impiega il modello pinhole (stenoscopio), per cui entrano in gioco i seguenti parametri: il centro della prospettiva (Oc): corrisponde al secondo punto nodale del sistema ottico; il piano immagine (p): il piano coincidente con lo schermo sul quale si forma l’immagine; l’asse principale: la retta coincidente con l’asse ottico (zc); il punto principale (OP), intersezione dell’asse ottico con il piano immagine; la distanza focale della telecamera (f): misura la distanza tra il centro della prospettiva ed il punto principale NOTA: nel sistema CCD la colonna corrisponde alla x, la riga alla y x OCCD y OP x p z f SC y OC

Reference System: From Camera to CCD x y z SRGB OCCD SCCD OC OP p 1 x0 y0 P K_rgb : intrinsic parameters matrix of RGB camera K_ir : intrinsic parameters matrix of IR camera fx , fy : focal length (in pixel) of the camera x0, y0: coordinate of principal point

Reference System: From Camera to CCD Homogeneous coordinate: x y z SRGB OCCD SCCD OC OP p 1 x0 y0 Pccd NOTE: the x coordinate in CCD reference system is the column and the y is the row Expressed in pixel Alredy compensated in the image given by Kinect

Refernece System: From IR to RGB x y z OP p R T

Microsoft Kinect: RGB image Distorsion alredy compensated

Things to do: Goal: find in 3d space the blue (or red) cube (dimension 10x10x10 cm ) Steps: Associate 3D points in IR sys with RGB info Identify ground plane and eliminate it (Ransac) Found the position of two cubes (SQ and Ransac) Chose the blue (or red) cube