Brilian T Nugraha

MAIN PAGE

TENSOR-RT PAGE

Webcam Inference with ONNXRuntime and TENSOR-RT

January 29th, 2022

The whole code is available at here.

After completing the Tensor-RT's Complete Inference, you will probably start itching to do some real stuff, aren't you? Btw, the webcam implementation is available at *onnx_trt_webcam.py*

Now, we’ll do something fun using IP-Camera using existing Android/Iphone app + Python!

What will be required in here:

  1. A Linux/Windows 10 PC (I am currently using Windows 10).
  2. Clone my repo, and you will realize that both previous blogs are connected to this one (as I am re-using those existing codes!).
  3. A phone with IOS/Android that has been installed with IP-Camera. I personally prefer to use DroidCam. I’ve compared “IP Wecam” and “DroidCam”, and IMHO, the first one seems to be laggy compared to the latter one.

Well, That’s all you need!

Implementation

Tensor-RT (abbrev. TRT)
ONNX-Runtime (abbrev. ORT)

For the implementation parts. I will separate the process into:

  1. Base variable initialization.
  2. Model initialization (either ORT or TRT).
  3. Webcam inference.
    • Capturing Ip Video Stream (cv2.VideoCapture).
    • Check if the Ip Camera is open/running.
    • Get stream/frame image from the Ip Camera.
    • Do Inference.
    • Draw output and FPS.
    • Close when the Ip Camera is closed or we manually close the streaming window (use ‘esc’/’q’ for exit).

Result

As I mentioned in my previous post, the winner of this test is Tensor-RT FP16 with ~17ms. In this test, the speed rank is also similar (TRT FP16 > TRT FP32 > ORT FP32 > ORT FP16). The difference is that the speed with TRT-FP16 is slightly slower when I tested it in using the WebcamViewer code.

Here are some snapshot between ONNX FP32 vs ONNX FP16 vs Tensor-RT FP32 vs Tensor-RT FP16.

ONNX FP32 ONNX FP16
Tensor-RT FP32 Tensor-RT FP16

As you can see, the models havbe is as the following:

Summary

In summary, the streaming speed using the Webcam Viewer script is as follow:

  1. Both Tensor-RT FP16 and FP32 could achieve ~30-33 FPS.
  2. ONNXRuntime FP32 with could achieve ~23-24 FPS.
  3. ONNXRuntime FP16 with could achieve ~21-22 FPS.

How to Reproduce

For starter, you can use this command to check the available options.

git clone my github (briliantnugraha.github.io)
cd ./all_rt/webcam_ort_trt
python ./codes/onnx_trt_yolox_webcam.py -h

And these are the command that I use to reproduce the streaming results (ORT and TRT)

python ./codes/onnx_trt_yolox_webcam.py --model_path "F:/gitdata/test_trt/yolox_m.trt" --rtsp_url "http://gyofarras:gyofarras@192.168.1.101:4747/video" #TRT-FP32
python ./codes/onnx_trt_yolox_webcam.py --model_path \"F:/gitdata/test_trt/yolox_m16.trt\" --rtsp_url \"http://gyofarras:gyofarras@192.168.1.101:4747/video" #TRT-FP16
python ./codes/onnx_trt_yolox_webcam.py --model_path \"F:/gitdata/test_trt/yolox_m.onnx\" --rtsp_url \"http://gyofarras:gyofarras@192.168.1.101:4747/video" #ORT-FP32
python ./codes/onnx_trt_yolox_webcam.py --model_path "F:/gitdata/test_trt/yolox_m16.onnx" --rtsp_url "http://gyofarras:gyofarras@192.168.1.101:4747/video" #ORT-FP16

There you go!

Hope this explanation helps. If there is any question or mistake with the content, please don’t hesitate to let me know, see you in the next blog and stay safe!

This site is open source. Improve this page.