OpenNI2 small depth map in upper left corner or without depth map at all

[ljacho]

I installed Ubuntu 12.04 on usb stick for purposes of my project. Libfreenect is installed correctly, glview works well. I have just copied libfreenectDriver.so to OpenNI-Linux-86.2.2/Samples/Bin/OpenNI2/Drivers and if I want to try SimpleViewer to see output from depth camera, it's only shown in upper left corner.
It works on another computer if I plug in some extra devices to usb ports like another usb stick, usb keyboard, external hardrive etc. If I unplug this "extra" device I don't get any output from depth camera or map is visible in corner. - I found out accidentally.

How can I solve this strange problem? How can I avoid having another usb device connected? On laptop I am using right now connecting another device doesn't help neither. I tryed to rebuild freenect, reinstall but without success.

Thank you for quick answers.

[piedar]

This is indeed very strange. Set Verbosity=0 in OpenNI.ini and post the terminal output from SimpleViewer.

What exactly do you mean by "only shown in upper left corner"? Do you see the whole depth image shrunk down or just a portion of the image? Does the color stream work OK? A screenshot would be helpful. Also, what is your CPU?

[ljacho]

OK, I made several experiments on different configurations with your suggestions, there are results:

TEST 1

OS: ubuntu 12.04
installed on: USB stick
configuration: desktop

usb connected devices:
kinect,usb stick(with currently booted OS), usb mouse, memory card reader (keyboard is connected per ps2)

DEPTH image doesnt work:

RGB image works:

logs with different Verbosity levels (SimpleViewer):

verbosity=0

student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 209 INFO New log started on 2014-02-25 14:12:32 249 INFO --- Filter Info --- Minimum Severity: VERBOSE 289 VERBOSE No override device in configuration file 316 VERBOSE Configuration has been read from './OpenNI.ini' 331 VERBOSE OpenNI 2.2.0 (Build 33)-Linux-x86 (Nov 12 2013 16:14:38) 345 VERBOSE Using './OpenNI2/Drivers' as driver path 358 VERBOSE Looking for drivers in drivers repository './OpenNI2/Drivers' 2897 INFO Device connected: Microsoft Kinect (freenect://0) 2953 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 5122 INFO New log started on 2014-02-25 14:12:32 5166 INFO --- Filter Info --- Minimum Severity: VERBOSE 5194 VERBOSE Initializing USB... 5308 INFO USB is initialized. 12941 INFO New log started on 2014-02-25 14:12:32 12986 INFO --- Filter Info --- Minimum Severity: VERBOSE 13016 VERBOSE Initializing USB... 13139 INFO USB is initialized. After initialization: 21821 VERBOSE Trying to open device by URI '(NULL)'

verbosity=1

student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 205 INFO New log started on 2014-02-25 14:13:27 252 INFO --- Filter Info --- Minimum Severity: INFO 2750 INFO Device connected: Microsoft Kinect (freenect://0) 2795 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 5173 INFO New log started on 2014-02-25 14:13:27 5191 INFO --- Filter Info --- Minimum Severity: VERBOSE 5310 INFO USB is initialized. 13175 INFO New log started on 2014-02-25 14:13:27 13221 INFO --- Filter Info --- Minimum Severity: VERBOSE 13363 INFO USB is initialized. After initialization:

verbosity=2

student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 207 INFO New log started on 2014-02-25 14:14:31 249 INFO --- Filter Info --- Minimum Severity: WARNING After initialization:

verbosity=3

student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 214 INFO New log started on 2014-02-25 14:14:53 255 INFO --- Filter Info --- Minimum Severity: ERROR After initialization:

usb connected devices:
kinect, usb stick(with currently booted OS), usb mouse, memory card reader, usb keyboard - can be changed with another usb stick or another usb device

DEPTH image works:

RGB image works:

logs with different Verbosity levels (SimpleViewer):

verbosity=0
student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Sples/Bin$ sudo ./SimpleViewer 208 INFO New log started on 2014-02-25 14:09:24 254 INFO --- Filter Info --- Minimum Severity: VERBOSE 296 VERBOSE No override device in configuration file 335 VERBOSE Configuration has been read from './OpenNI.ini' 357 VERBOSE OpenNI 2.2.0 (Build 33)-Linux-x86 (Nov 12 2013 16:14:38) 372 VERBOSE Using './OpenNI2/Drivers' as driver path 386 VERBOSE Looking for drivers in drivers repository './OpenNI2/Drivers' 3098 INFO Device connected: Microsoft Kinect (freenect://0) 3141 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 5354 INFO New log started on 2014-02-25 14:09:24 5385 INFO --- Filter Info --- Minimum Severity: VERBOSE 5410 VERBOSE Initializing USB... 5540 INFO USB is initialized. 13979 INFO New log started on 2014-02-25 14:09:24 14025 INFO --- Filter Info --- Minimum Severity: VERBOSE 14059 VERBOSE Initializing USB... 14180 INFO USB is initialized. After initialization: 20661 VERBOSE Trying to open device by URI '(NULL)'

verbosity=1
student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 202 INFO New log started on 2014-02-25 14:09:54 249 INFO --- Filter Info --- Minimum Severity: INFO 2492 INFO Device connected: Microsoft Kinect (freenect://0) 2520 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 3772 INFO New log started on 2014-02-25 14:09:54 3791 INFO --- Filter Info --- Minimum Severity: VERBOSE 3859 INFO USB is initialized. 8353 INFO New log started on 2014-02-25 14:09:54 8373 INFO --- Filter Info --- Minimum Severity: VERBOSE 8448 INFO USB is initialized. After initialization:
verbosity=2
student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 246 INFO New log started on 2014-02-25 14:10:16 291 INFO --- Filter Info --- Minimum Severity: WARNING After initialization: verbosity=3 student@student-ADATA:~/Desktop/kinect/OpenNI2zostranky/OpenNI-Linux-x86-2.2/Samples/Bin$ sudo ./SimpleViewer 203 INFO New log started on 2014-02-25 14:10:40 253 INFO --- Filter Info --- Minimum Severity: ERROR After initialization:

TEST 2

OS: ubuntu 12.04
installed on: USB stick
configuration: netbook

usb connected devices:
kinect, kinect,usb stick(with currently booted OS)

DEPTH image doesnt work:

RGB works:

verbosity 0 log:
572 INFO New log started on 2014-02-26 12:03:38 681 INFO --- Filter Info --- Minimum Severity: VERBOSE 797 VERBOSE No override device in configuration file 872 VERBOSE Configuration has been read from './OpenNI.ini' 916 VERBOSE OpenNI 2.2.0 (Build 33)-Linux-x86 (Nov 12 2013 16:14:38) 1004 VERBOSE Using './OpenNI2/Drivers' as driver path 1072 VERBOSE Looking for drivers in drivers repository './OpenNI2/Drivers' 30841 INFO Device connected: Microsoft Kinect (freenect://0) 30955 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 91307 INFO New log started on 2014-02-26 12:03:38 91393 INFO --- Filter Info --- Minimum Severity: VERBOSE 91455 VERBOSE Initializing USB... 91799 INFO USB is initialized. 156836 INFO New log started on 2014-02-26 12:03:38 156923 INFO --- Filter Info --- Minimum Severity: VERBOSE 156988 VERBOSE Initializing USB... 157412 INFO USB is initialized. After initialization: 198527 VERBOSE Trying to open device by URI '(NULL)' [Stream 70] Expected 1748 data bytes, but got 94

lsusb output:
Bus 001 Device 002: ID 125f:c82a A-DATA Technology Co., Ltd. Bus 001 Device 008: ID 0409:005a NEC Corp. HighSpeed Hub Bus 001 Device 003: ID 04f2:b26f Chicony Electronics Co., Ltd Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 010: ID 045e:02ad Microsoft Corp. Xbox NUI Audio Bus 001 Device 009: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor Bus 001 Device 011: ID 045e:02ae Microsoft Corp. Xbox NUI Camera

usb connected devices:
kinect, kinect,usb stick(with currently booted OS) + "extra device" usb harddrive

doesn't work neither..

DEPTH image doesnt work:

RGB works:

verbosity 0 log:
572 INFO New log started on 2014-02-26 12:03:38 681 INFO --- Filter Info --- Minimum Severity: VERBOSE 797 VERBOSE No override device in configuration file 872 VERBOSE Configuration has been read from './OpenNI.ini' 916 VERBOSE OpenNI 2.2.0 (Build 33)-Linux-x86 (Nov 12 2013 16:14:38) 1004 VERBOSE Using './OpenNI2/Drivers' as driver path 1072 VERBOSE Looking for drivers in drivers repository './OpenNI2/Drivers' 30841 INFO Device connected: Microsoft Kinect (freenect://0) 30955 INFO Device state changed: Microsoft Kinect (freenect://0) to 0 91307 INFO New log started on 2014-02-26 12:03:38 91393 INFO --- Filter Info --- Minimum Severity: VERBOSE 91455 VERBOSE Initializing USB... 91799 INFO USB is initialized. 156836 INFO New log started on 2014-02-26 12:03:38 156923 INFO --- Filter Info --- Minimum Severity: VERBOSE 156988 VERBOSE Initializing USB... 157412 INFO USB is initialized. After initialization: 198527 VERBOSE Trying to open device by URI '(NULL)'

lsusb output:
Bus 001 Device 002: ID 125f:c82a A-DATA Technology Co., Ltd. Bus 001 Device 008: ID 0409:005a NEC Corp. HighSpeed Hub Bus 001 Device 003: ID 04f2:b26f Chicony Electronics Co., Ltd Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Bus 001 Device 010: ID 045e:02ad Microsoft Corp. Xbox NUI Audio Bus 001 Device 009: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor Bus 001 Device 011: ID 045e:02ae Microsoft Corp. Xbox NUI Camera

thank you for your help, I would be glad if you could join me per teamviewer for example to find out more.. if needed. Thank you one more time! 👍

hope that this exhaustive comment will be helpfull for more people and we will solve this strange issue..

[piedar]

This is great information; thanks for your efforts!

I think I found the cause, and with any luck 62f0a0d will solve this for you as well.

[ljacho]

Thank you for quick answer.

This problem is partially solved because I can get it working with your advise but it still matters how many usb devices are connected - really don't know why.

I took updated source and build freenect when these usb devices were connected:
kinect, usb stick, usb keyboard, usb mouse
OS booted from SATA harddrive - ubuntu 13.04 32-bit.

depth map didn't work. If I disconnect 2 devices - it doesn't matter which ones - it works. I have to disconnect two. - I can't work with kinect with anything over count of 2 devices (e.g. one usb stick) connected.

If I connect kinect alone and disconnect all devices - it doesn't work neither. - 2 must be connected.

kind of disappointment..

In my opinion - there could be a problem in managing usb devices from OS's side (libusb?), or OpenNI (I tried several releases with same results - I am using OpenNI 2.2.0.33 Beta - release I forgot to mention), couldn't it?
I would like to say it's not a problem of freenect essentially..

Is there anything I should try?

[piedar]

I am running short on ideas...

1. You are using an external power source for the Kinect (i.e. it's plugged into the wall)?

2. I should see the libfreenect debug log in SimpleViewer output. To capture all output to a file, you must redirect stderr:

   ./SimpleViewer > output.txt 2>&1
   

3. We should see if the driver is receiving depth frames from libfreenect:

diff --git a/OpenNI2-FreenectDriver/src/DepthStream.cpp b/OpenNI2-FreenectDriver/src/DepthStream.cpp
index f7addca..1711c60 100644
--- a/OpenNI2-FreenectDriver/src/DepthStream.cpp
+++ b/OpenNI2-FreenectDriver/src/DepthStream.cpp
@@ -54,6 +54,11 @@ OniStatus DepthStream::setVideoMode(OniVideoMode requested_mode)

 void DepthStream::populateFrame(void* data, OniFrame* frame) const
 {
+  if (frame->frameIndex % 60 == 0)
+  {
+    WriteMessage("DepthStream::populateFrame() received 60 frames");
+  }
+
   frame->sensorType = sensor_type;
   frame->stride = video_mode.resolutionX * sizeof(uint16_t);

If that message shows up in the output, the problem is probably in OpenNI2. If not, the problem is probably in libusb, libfreenect, or elsewhere in the driver.

[ljacho]

1. Yes, kinect is plugged for sure.
2. There are logfiles and here is conclusion:
   (OpenNI2 from webside + latest libfreenect)

output file	number of usb devices (kinect isn't involved in count)	works?	WriteMessage appeared?
output.txt	0	yes	yes
output-1more.txt	1	yes	yes
output-2more.txt	2	yes	yes
output-3more.txt	3	no	yes
output-4more.txt	4	no	yes

It seems it is really random!

I have possibility to try it on more computers, with 2 different kinect devices.. I tried several releases and also source from github for both - freenect and OpenNI2. Ubuntu OS booted from usb stick (because of my need - portability)/HDD. And it's still random.. one should say I am mad (lol) but I really wonder why anybody is not complaining about this strange thing..

...so do you think freenect is good, and OpenNI2 is bad? What's your conclusion?

[piedar]

From all this evidence, my best guess is that due to USB bandwidth. The Kinect uses quite a lot and it's likely that as more devices are added, less bandwidth goes to the Kinect. I think this is supported by your output logs - generally, more devices == more dropped frames.

This doesn't explain why freenect-glview works and SimpleViewer doesn't, but maybe the extra overhead of a buffer copy causes even more dropped frames. Less likely but quite possibly, it could be some hidden bug in the C++ wrapper libfreenect.hpp or my driver.

That said, I still can't replicate the issue on my machine, even with all 6 USB ports filled.

[piedar]

PS: I'm on vacation next week, but I haven't given up on this issue!

[ljacho]

I appreciate your interest in my issue a lot. As I said anybody else is not complaining.. You couldn't replicate it and I can everytime I want.. I am open to resolve it and help how I can. Anyway enjoy your vacation and then we could back to business! Just let me know.. I will write personal message.

[lnegrol]

Hello,
I have read and would like to tell you that I have installed Ubuntu 12.04.03 LTS (32 bits) in a Toshiba Satellite L755-1J9 with the following components:

• Intel® Core™ i5-2450M (2.50 / 3.10 Turbo GHz, FSB: 1333 MHz, 3 MB Cache)
• Samsung 16 GB (8 GB x 2) DDR3 RAM (1,333 MHz) Dual Channel
• Samsung SSD 840 EVO 120 GB (SATA 6Gb/s compatible with 3Gb/s)
• Toshiba SATA 640 GB 5,400 rpm
• NVIDIA® GeForce® GT 520M with CUDA technology (1GB DDR3 VRAM)
• 2x USB 2.0 and 1x USB 3.0

I tested the kinect works perfectly on all usb ports.

Right now I'm busy with work, but when I have a hole, I try to install Ubuntu on a USB.

A greeting.

[piedar]

@sbtgE You might experiment with the values of PKTS_PER_XFER and NUM_XFERS in usb_libusb10.h. I've seen decent results with

#define PKTS_PER_XFER 128
#define NUM_XFERS 6

[piedar]

#378 addresses performance and may help in this case.

If it helps but not enough, you could try more aggressive optimizations:

cmake .. -L -DCMAKE_C_FLAGS="-march=native -O3" -DCMAKE_CXX_FLAGS="-march=native -O3"

Note that binaries produced may not be portable to different CPUs.