Merge branch 'NVIDIA-AI-IOT:master' into master

Author: benaz14

.gitmodules

@@ -1,6 +1,10 @@
-[submodule "3rdparty/pybind11"]
-	path = 3rdparty/pybind11
+[submodule "bindings/3rdparty/pybind11"]
+	path = bindings/3rdparty/pybind11
 	url = https://github.com/pybind/pybind11.git
-[submodule "3rdparty/gst-python"]
-	path = 3rdparty/gst-python
-	url = https://github.com/GStreamer/gst-python.git
+[submodule "bindings/3rdparty/git-partial-submodule"]
+	path = bindings/3rdparty/git-partial-submodule
+	url = https://github.com/Reedbeta/git-partial-submodule.git
+[submodule "bindings/3rdparty/gstreamer"]
+	path = bindings/3rdparty/gstreamer
+	url = https://github.com/GStreamer/gstreamer.git
+	sparse-checkout = subprojects/gst-python/

3rdparty/gst-python

@@ -1,15 +1,55 @@
 # Frequently Asked Questions and Troubleshooting Guide
 
-* [Git clone fails due to Gstreamer repo access error](#faq9)
-* [Application fails to work with mp4 stream](#faq0)  
-* [Ctrl-C does not stop the app during engine file generation](#faq1)  
-* [Application fails to create gst elements](#faq2)  
-* [GStreamer debugging](#faq3)  
-* [Application stuck with no playback](#faq4)  
-* [Error on setting string field](#faq5)  
-* [Pipeline unable to perform at real time](#faq6)  
-* [Triton container problems with multi-GPU setup](#faq7)
-* [ModuleNotFoundError: No module named 'pyds'](#faq8)
+- [Frequently Asked Questions and Troubleshooting Guide](#frequently-asked-questions-and-troubleshooting-guide)
+    - [App causes Jetson Orin Nano to freeze and reboot](#app-causes-jetson-orin-nano-to-freeze-and-reboot)
+    - [Using new gst-nvstreammux](#using-new-gst-nvstreammux)
+    - [Git clone fails due to Gstreamer repo access error](#git-clone-fails-due-to-gstreamer-repo-access-error)
+    - [Application fails to work with mp4 stream](#application-fails-to-work-with-mp4-stream)
+    - [Ctrl-C does not stop the app during engine file generation](#ctrl-c-does-not-stop-the-app-during-engine-file-generation)
+    - [Application fails to create gst elements](#application-fails-to-create-gst-elements)
+    - [GStreamer debugging](#gstreamer-debugging)
+    - [Application stuck with no playback](#application-stuck-with-no-playback)
+    - [Error on setting string field](#error-on-setting-string-field)
+    - [Pipeline unable to perform at real time](#pipeline-unable-to-perform-at-real-time)
+    - [Triton container problems with multi-GPU setup](#triton-container-problems-with-multi-gpu-setup)
+    - [ModuleNotFoundError: No module named 'pyds'](#modulenotfounderror-no-module-named-pyds)
+
+<a name="faq11"></a>
+### App causes Jetson Orin Nano to freeze and reboot
+Most of the DeepStream Python apps are written to use hardware encoders. The Jetson Orin Nano does not have any hardware encoders, so the app must be modified to use software encoders in the pipeline. Please see [deepstream-test1-rtsp-out](./apps/deepstream-test1-rtsp-out/deepstream_test_1_rtsp_out.py):
+```python
+if codec == "H264":
+  encoder = Gst.ElementFactory.make("nvv4l2h264enc", "encoder")
+  if enc_type == 0: # HW encoder
+      encoder = Gst.ElementFactory.make("nvv4l2h264enc", "encoder")
+  else: # SW encoder
+      encoder = Gst.ElementFactory.make("x264enc", "encoder")
+```
+
+<a name="faq10"></a>
+### Using new gst-nvstreammux
+Most DeepStream Python apps are written to use the default nvstreammux and have lines of code written to set nvstreammux properties which are deprecated in the new gst-nvstreammux. See the [DeepStream documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux2.html) for more information on the new gst-nvstreammux plugin. To use the new nvstreammux, set the `USE_NEW_NVSTREAMMUX` environment variable before running the app. For example:
+```bash
+   $ export USE_NEW_NVSTREAMMUX="yes"
+   $ python3 deepstream_test_1.py ../../../../samples/streams/sample_720p.h264
+```
+The app itself must be modified not to set deprecated properties when using the new nvstreammux. See [deepstream-test1](./apps/deepstream-test1/deepstream_test_1.py):
+```python
+if os.environ.get('USE_NEW_NVSTREAMMUX') != 'yes': # Only set these properties if not using new gst-nvstreammux
+  streammux.set_property('width', 1920)
+  streammux.set_property('height', 1080)
+  streammux.set_property('batched-push-timeout', 4000000)
+```
+
+Running apps without this modification will result in such an error:
+```
+Traceback (most recent call last):
+  File "deepstream_test_1.py", line 255, in <module>
+    sys.exit(main(sys.argv))
+  File "deepstream_test_1.py", line 194, in main
+    streammux.set_property('width', 1920)
+TypeError: object of type `GstNvStreamMux' does not have property `width'
+```
 
 <a name="faq9"></a>
 ### Git clone fails due to Gstreamer repo access error
@@ -43,9 +83,9 @@ https://docs.python.org/3/library/signal.html
 To work around this:  
 1. Use ctrl-z to bg the process  
 2. Optionally run "jobs" if there are potentially multiple bg processes:  
-    $ jobs  
-    [1]-  Stopped                 python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264  (wd: /opt/nvidia/deepstream/deepstream-4.0/sources/apps/python/deepstream-test1)  
-    [2]+  Stopped                 python3 deepstream_test_2.py /opt/nvidia/deepstream/deepstream-4.0/samples/streams/sample_720p.h264  
+    $ jobs
+    [1]-  Stopped                 python3 deepstream_test_1.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264  (wd: /opt/nvidia/deepstream/deepstream-4.0/sources/apps/python/deepstream-test1)
+    [2]+  Stopped                 python3 deepstream_test_2.py /opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.h264
 3. Kill the bg job:  
     $ kill %<job number, 1 or 2 from above. e.g. kill %1>  
 
@@ -125,5 +165,5 @@ The pyds wheel installs the pyds.so library where all the pip packages are store
 
 Command to install the pyds wheel is:
 ```bash
-   $ pip3 install ./pyds-1.1.0-py3-none*.whl
-```
+   $ pip3 install ./pyds-1.2.0*.whl
+```

3rdparty/pybind11

@@ -15,10 +15,10 @@ This guide provides resources for DeepStream application development in Python.
 <a name="prereqs"></a>
 ## Prerequisites
 
-* Ubuntu 20.04
-* [DeepStream SDK 6.1.1](https://developer.nvidia.com/deepstream-download) or later
-* Python 3.8
-* [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.16.2
+* Ubuntu 22.04
+* [DeepStream SDK 7.1](https://developer.nvidia.com/deepstream-download) or later
+* Python 3.10
+* [Gst Python](https://gstreamer.freedesktop.org/modules/gst-python.html) v1.20.3
 
 Gst python should be already installed on Jetson.  
 If missing, install with the following steps:
@@ -46,11 +46,11 @@ Note: Compiling bindings now also generates a pip installable python wheel for t
 <a name="run_samples"></a>
 ## Running Sample Applications
 
-Clone the deepstream_python_apps repo under <DeepStream 6.1.1 ROOT>/sources:
+Clone the deepstream_python_apps repo under <DeepStream ROOT>/sources:
 git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps
 
 This will create the following directory:  
-```<DeepStream 6.1.1 ROOT>/sources/deepstream_python_apps```
+```<DeepStream ROOT>/sources/deepstream_python_apps```
 
 The Python apps are under the "apps" directory.  
 Go into each app directory and follow instructions in the README.  
@@ -84,12 +84,12 @@ Memory for MetaData is shared by the Python and C/C++ code paths. For example, a
 
 #### Allocations
 
-When MetaData objects are allocated in Python, an allocation function is provided by the bindings to ensure proper memory ownership of the object. If the constructor is used, the the object will be claimed by the garbage collector when its Python references terminate. However, the object will still need to be accessed by C/C++ code downstream, and therefore must persist beyond those Python references.  
+When MetaData objects are allocated in Python, an allocation function is provided by the bindings to ensure proper memory ownership of the object. If the constructor is used, the object will be claimed by the garbage collector when its Python references terminate. However, the object will still need to be accessed by C/C++ code downstream, and therefore must persist beyond those Python references.  
 
 Example:
 To allocate an NvDsEventMsgMeta instance, use this:  
 ```python
-msg_meta = pyds.alloc_nvds_event_msg_meta() # get reference to allocated instance without claiming memory ownership  
+msg_meta = pyds.alloc_nvds_event_msg_meta(user_event_meta) # get reference to allocated instance without claiming memory ownership  
 ```
 NOT this:  
 ```python
@@ -173,16 +173,21 @@ frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
 
 Custom MetaData added to NvDsUserMeta require custom copy and release functions. The MetaData library relies on these custom functions to perform deep-copy of the custom structure, and free allocated resources. These functions are registered as callback function pointers in the NvDsUserMeta structure.
 
-Callback functions are registered using these functions:  
+##### Deprecation Warning
+Previously, Callback functions were registered using these functions:  
 ```python
 pyds.set_user_copyfunc(NvDsUserMeta_instance, copy_function)
 pyds.set_user_releasefunc(NvDsUserMeta_instance, free_func)
 ```
 
-*NOTE*: Callbacks need to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
+These are now DEPRECATED and are replaced by similar implementation in the binding itself. These are event_msg_meta_copy_func() and event_msg_meta_release_func() respectively. These can be found inside [bindschema.cpp](bindings/src/bindschema.cpp)
+
+##### Deprecation Warning
+*NOTE*: Previously, callbacks needed to be unregistered with the bindings library before the application exits. The bindings library currently keeps global references to the registered functions, and these cannot last beyond bindings library unload which happens at application exit. Use the following function to unregister all callbacks:
 ```pyds.unset_callback_funcs()```  
+These callbacks are automatically set inside the alloc_nvds_event_msg_meta() function and should NOT be set from the python application (e.g. deepstream-test4)
 
-See the deepstream-test4 sample application for an example of callback registration and unregistration.  
+The deepstream-test4 sample application has been updated to show an example of removal of these callback registration and unregistration.
 
 Limitation: the bindings library currently only supports a single set of callback functions for each application. The last registered function will be used.  
 
@@ -212,5 +217,5 @@ This function populates the input buffer with a timestamp generated according to
 <a name="imagedata_access"></a>
 ## Image Data Access
 
-Decoded images are accessible as NumPy arrays via the `get_nvds_buf_surface` function. This function is documented in the [API Guide](https://docs.nvidia.com/metropolis/deepstream/6.0.1/python-api/index.html).
+Decoded images are accessible as NumPy arrays via the `get_nvds_buf_surface` function. This function is documented in the [API Guide](https://docs.nvidia.com/metropolis/deepstream/dev-guide/python-api/index.html).
 Please see the [deepstream-imagedata-multistream](apps/deepstream-imagedata-multistream) sample application for an example of image data usage.

FAQ.md

@@ -2,20 +2,35 @@
 
 This repository contains Python bindings and sample applications for the [DeepStream SDK](https://developer.nvidia.com/deepstream-sdk).  
 
-SDK version supported: 6.1.1
+SDK version supported: 7.1
 
-<b>The bindings sources along with build instructions are now available under [bindings](bindings)!  </b>
+<b>This release only supports Ubuntu 22.04 for DeepStreamSDK 7.1 with Python 3.10 and [gst-python](3rdparty/gst-python/) 1.20.3! Ubuntu 20.04 for DeepStreamSDK 6.3 with Python 3.8 support is NOW DEPRECATED</b>
 
-<b>This release comes with Operating System upgrades (from Ubuntu 18.04 to Ubuntu 20.04) for DeepStreamSDK 6.1.1 support. This translates to upgrade in Python version to 3.8 and [gst-python](3rdparty/gst-python/) version has also been upgraded to 1.16.2 !</b>
-
-Download the latest release package complete with bindings and sample applications from the [release section](../../releases).  
+The bindings sources along with build instructions are available under [bindings](bindings)! PyDS 1.2.0 for DeepStream 7.1 uses an updated build system for PyPA to support pip 24.2, which deprecated setup.py command line. We include one [guide](bindings/BINDINGSGUIDE.md) for contributing to bindings and another [guide](bindings/CUSTOMUSERMETAGUIDE.md) for advanced use-cases such as writing bindings for custom data structures. 
 
 Please report any issues or bugs on the [DeepStream SDK Forums](https://devtalk.nvidia.com/default/board/209). This enables the DeepStream community to find help at a central location.
 
+<b>NOTE:<b> numpy 2.x is not supported. If you have numpy 2.x installed, please downgrade by running
+```
+pip3 install --force-reinstall numpy==1.26.0
+```
+
+<b>NOTE:<b> deepstream-segmask and deepstream-segmentation applications are not currently supported by DeepStream 7.1, due to removal of segmentation models.
+
+<b>NOTE for DeepStream dockers:<b> If you installed PyDS by running the user_deepstream_python_apps_install.sh script, be sure you also run the <b>user_additional_install.sh script<b>.
+
 - [DeepStream Python Apps](#deepstream-python-apps)
+  - [Setup](#setup)
   - [Python Bindings](#python-bindings)
+    - [Python Bindings Breaking API Change](#python-bindings-breaking-api-change)
   - [Sample Applications](#sample-applications)
 
+## Setup
+Once you have DeepStreamSDK pre-requisites and DeepStreamSDK installed on the system, navigate to <DS_ROOT>/sources/ dir which is /opt/nvidia/deepstream/deepstream/sources/ and git clone deepstream_python_apps repo here.
+
+The latest bindings can be installed from [release section](../../releases).
+You can also build the bindings from source using the instructions in the [bindings readme](bindings/README.md) if needed.
+
 <a name="metadata_bindings"></a>
 ## Python Bindings
 
@@ -28,6 +43,9 @@ Python [bindings](bindings) are provided as part of this repository. This module
 
 These bindings support a Python interface to the MetaData structures and functions. Usage of this interface is documented in the [HOW-TO Guide](HOWTO.md) and demonstrated in the sample applications.  
 
+### Python Bindings Breaking API Change
+The binding for function alloc_nvds_event_msg_meta() now expects a NvDsUserMeta pointer which the NvDsEventMsgMeta is associated with. Please refer to [deepstream-test4](apps/deepstream-test4) and [bindschema.cpp](bindings/src/bindschema.cpp) for reference.
+
 <a name="sample_applications"></a>
 ## Sample Applications
 
@@ -41,22 +59,25 @@ To run the sample applications or write your own, please consult the [HOW-TO Gui
 </p>
 
 We currently provide the following sample applications:
-* [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline
+* [deepstream-test1](apps/deepstream-test1) -- 4-class object detection pipeline, also demonstrates support for new nvstreammux
 * [deepstream-test2](apps/deepstream-test2) -- 4-class object detection, tracking and attribute classification pipeline
-* [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection - now also supports triton inference server, no-display mode, file-loop and silent mode
+* [deepstream-test3](apps/deepstream-test3) -- multi-stream pipeline performing 4-class object detection, also supports triton inference server, no-display mode, file-loop and silent mode
 * [deepstream-test4](apps/deepstream-test4) -- msgbroker for sending analytics results to the cloud
 * [deepstream-imagedata-multistream](apps/deepstream-imagedata-multistream) -- multi-stream pipeline with access to image buffers
 * [deepstream-ssd-parser](apps/deepstream-ssd-parser) -- SSD model inference via Triton server with output parsing in Python
-* [deepstream-test1-usbcam](apps/deepstream-test1-usbcam) -- deepstream-test1 pipelien with USB camera input
-* [deepstream-test1-rtsp-out](apps/deepstream-test1-rtsp-out) -- deepstream-test1 pipeline with RTSP output
+* [deepstream-test1-usbcam](apps/deepstream-test1-usbcam) -- deepstream-test1 pipeline with USB camera input
+* [deepstream-test1-rtsp-out](apps/deepstream-test1-rtsp-out) -- deepstream-test1 pipeline with RTSP output, demonstrates adding software encoder option to support Jetson Orin Nano
 * [deepstream-opticalflow](apps/deepstream-opticalflow) -- optical flow and visualization pipeline with flow vectors returned in NumPy array
-* [deepstream-segmentation](apps/deepstream-segmentation) -- segmentation and visualization pipeline with segmentation mask returned in NumPy array
+* [deepstream-segmentation](apps/deepstream-segmentation) -- **NOT CURRENTLY SUPPORTED IN DS 7.1** segmentation and visualization pipeline with segmentation mask returned in NumPy array
 * [deepstream-nvdsanalytics](apps/deepstream-nvdsanalytics) -- multistream pipeline with analytics plugin
 * [runtime_source_add_delete](apps/runtime_source_add_delete) -- add/delete source streams at runtime
 * [deepstream-imagedata-multistream-redaction](apps/deepstream-imagedata-multistream-redaction) -- multi-stream pipeline with face detection and redaction
-* [deepstream-rtsp-in-rtsp-out](apps/deepstream-rtsp-in-rtsp-out) -- multi-stream pipeline with RTSP input/output
+* [deepstream-rtsp-in-rtsp-out](apps/deepstream-rtsp-in-rtsp-out) -- multi-stream pipeline with RTSP input/output - has command line option "--rtsp-ts" for configuring the RTSP source to attach the timestamp rather than the streammux
 * [deepstream-preprocess-test](apps/deepstream-preprocess-test) -- multi-stream pipeline using nvdspreprocess plugin with custom ROIs
-* <b>NEW</b> [deepstream-demux-multi-in-multi-out](apps/deepstream-demux-multi-in-multi-out) -- multi-stream pipeline using nvstreamdemux plugin to generated separate buffer outputs
+* [deepstream-demux-multi-in-multi-out](apps/deepstream-demux-multi-in-multi-out) -- multi-stream pipeline using nvstreamdemux plugin to generated separate buffer outputs
+* [deepstream-imagedata-multistream-cupy](apps/deepstream-imagedata-multistream-cupy) -- access imagedata buffer from GPU in a multistream source as CuPy array - x86 only
+* [deepstream-segmask](apps/deepstream-segmask) -- **NOT CURRENTLY SUPPORTED IN DS 7.1** access and interpret segmentation mask information from NvOSD_MaskParams
+* [deepstream-custom-binding-test](apps/deepstream-custom-binding-test) -- demonstrate usage of NvDsUserMeta for attaching custom data structure - see also the [Custom User Meta Guide](bindings/CUSTOMUSERMETAGUIDE.md)
 
 
 Detailed application information is provided in each application's subdirectory under [apps](apps).